Nov 02 09:32:22.610543 ip-10-0-138-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 09:32:22.610556 ip-10-0-138-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 09:32:22.610565 ip-10-0-138-234 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 09:32:22.610813 ip-10-0-138-234 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 09:32:32.754763 ip-10-0-138-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 09:32:32.754779 ip-10-0-138-234 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 53e63310996b479fa3e82b690d0f9b97 -- Nov 02 09:34:40.604223 ip-10-0-138-234 systemd[1]: Starting Kubernetes Kubelet... Nov 02 09:34:40.998123 ip-10-0-138-234 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 09:34:40.998123 ip-10-0-138-234 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 09:34:40.998123 ip-10-0-138-234 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 09:34:40.998123 ip-10-0-138-234 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 09:34:41.002717 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002628 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:34:41.002717 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002717 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:34:41.002717 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002721 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002725 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002728 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002731 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002734 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002737 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002740 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002743 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002745 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002748 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002751 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002753 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002760 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002763 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002766 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002768 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002771 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002774 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002777 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:34:41.002821 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002780 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002783 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002785 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002788 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002791 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002793 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002796 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002800 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002805 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002809 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002812 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002815 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002819 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002822 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002825 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002827 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002830 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002833 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002836 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:34:41.003283 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002838 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002841 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002844 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002846 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002849 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002853 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002857 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002860 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002863 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002865 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002868 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002871 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002873 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002876 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002879 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002882 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002885 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002900 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002902 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:34:41.003783 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002905 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002908 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002911 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002914 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002916 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002919 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002922 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002925 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002927 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002930 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002933 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002935 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002938 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002941 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002943 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002946 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002948 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002951 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002953 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002956 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:34:41.004255 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002958 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002962 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002964 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002967 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002969 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002972 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002974 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002977 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002980 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002982 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002985 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002988 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002991 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002994 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.002997 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003000 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003002 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003005 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003008 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003011 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:34:41.004729 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003013 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003016 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003018 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003020 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003023 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003027 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003030 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003873 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003880 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003884 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003903 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003906 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003909 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003912 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003915 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003918 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003921 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003924 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003926 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003929 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:34:41.005206 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003932 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003934 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003938 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003941 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003944 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003947 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003949 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003952 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003954 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003957 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003960 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003964 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003970 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003973 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003976 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003980 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003982 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003985 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003987 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:34:41.005680 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003990 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003993 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003996 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.003998 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004001 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004003 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004009 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004012 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004016 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004019 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004021 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004024 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004027 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004029 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004032 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004035 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004039 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004042 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004044 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004047 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:34:41.006154 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004049 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004052 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004055 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004057 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004060 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004063 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004066 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004068 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004071 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004073 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004076 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004078 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004081 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004084 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004086 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004089 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004091 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004094 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004098 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004101 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:34:41.006623 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004103 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004106 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004109 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004111 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004114 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004117 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004120 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004122 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004125 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004128 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004131 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004133 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004136 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004138 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004141 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004145 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004149 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004152 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004155 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:34:41.007360 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004158 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004160 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004163 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004165 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004167 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004171 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004174 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004176 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004179 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004181 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004184 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004188 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004190 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004193 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004196 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004695 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004700 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004703 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004706 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:34:41.008091 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004709 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004712 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004716 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004719 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004721 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004724 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004726 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004729 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004732 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004735 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004738 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004741 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004743 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004746 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004749 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004751 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004754 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004757 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004759 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:34:41.008647 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004763 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004768 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004772 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004775 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004778 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004781 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004783 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004786 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004788 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004792 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004794 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004797 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004799 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004801 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004804 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004807 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004810 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004812 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004814 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:34:41.009137 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004817 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004819 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004822 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004827 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004830 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004832 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004835 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004837 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004840 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004842 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004844 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004847 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004849 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004852 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004854 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004857 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004859 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004862 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004865 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:34:41.009616 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004868 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004871 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004873 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004876 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004879 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004882 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004884 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004901 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004904 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004907 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004910 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004913 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004915 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004918 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004921 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004924 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004927 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004929 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004932 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004934 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:34:41.010083 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004937 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004939 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004942 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004944 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004947 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004949 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004952 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004955 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004957 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004960 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004962 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004965 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004968 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004971 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004973 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004976 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004979 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004982 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004984 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:34:41.010566 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004987 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004989 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004992 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004995 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.004997 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.005001 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006044 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006060 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006068 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006073 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006078 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006082 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006087 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006092 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006096 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006100 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006103 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006107 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006111 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006114 2581 flags.go:64] FLAG: --cgroup-root="" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006117 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006120 2581 flags.go:64] FLAG: --client-ca-file="" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006123 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 02 09:34:41.011081 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006126 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006131 2581 flags.go:64] FLAG: --cluster-domain="" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006134 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006137 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006140 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006144 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006148 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006151 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006155 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006159 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006162 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006166 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006169 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006172 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006175 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006180 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006183 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006186 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006189 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006192 2581 flags.go:64] FLAG: --enable-server="true" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006195 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006206 2581 flags.go:64] FLAG: --event-burst="100" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006209 2581 flags.go:64] FLAG: --event-qps="50" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006212 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 09:34:41.011627 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006215 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006219 2581 flags.go:64] FLAG: --eviction-hard="" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006223 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006228 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006232 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006235 2581 flags.go:64] FLAG: --eviction-soft="" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006240 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006243 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006246 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006249 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006252 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006255 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006258 2581 flags.go:64] FLAG: --feature-gates="" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006262 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006266 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006269 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006273 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006277 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006280 2581 flags.go:64] FLAG: --help="false" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006284 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006287 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006291 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006294 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 09:34:41.012224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006297 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006301 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006304 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006307 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006310 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006313 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006316 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006319 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006322 2581 flags.go:64] FLAG: --kube-reserved="" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006325 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006328 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006331 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006336 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006339 2581 flags.go:64] FLAG: --lock-file="" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006342 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006346 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006349 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006355 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006358 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006361 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006365 2581 flags.go:64] FLAG: --logging-format="text" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006368 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006371 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006374 2581 flags.go:64] FLAG: --manifest-url="" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006377 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 02 09:34:41.012769 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006382 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006385 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006390 2581 flags.go:64] FLAG: --max-pods="110" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006393 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006397 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006400 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006403 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006406 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006409 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006412 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006420 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006423 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006426 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006429 2581 flags.go:64] FLAG: --pod-cidr="" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006433 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006436 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006439 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006443 2581 flags.go:64] FLAG: --port="10250" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006445 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006450 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01995ce47ff987087" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006453 2581 flags.go:64] FLAG: --qos-reserved="" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006456 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006461 2581 flags.go:64] FLAG: --register-node="true" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006464 2581 flags.go:64] FLAG: --register-with-taints="" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006468 2581 flags.go:64] FLAG: --registry-burst="10" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006471 2581 flags.go:64] FLAG: --registry-qps="5" Nov 02 09:34:41.013418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006474 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006477 2581 flags.go:64] FLAG: --reserved-memory="" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006482 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006486 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006489 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006492 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006495 2581 flags.go:64] FLAG: --runonce="false" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006498 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006502 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006505 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006508 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006511 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006515 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006518 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006521 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006524 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006527 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006530 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006533 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006536 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006539 2581 flags.go:64] FLAG: --system-cgroups="" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006542 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006548 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006551 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006553 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 09:34:41.014048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006559 2581 flags.go:64] FLAG: --tls-min-version="" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006562 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006565 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006570 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006573 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006576 2581 flags.go:64] FLAG: --v="2" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006581 2581 flags.go:64] FLAG: --version="false" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006585 2581 flags.go:64] FLAG: --vmodule="" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006589 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.006592 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008706 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008735 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008740 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008745 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008749 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008754 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008758 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008762 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008766 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008770 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008775 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:34:41.014681 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008781 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008788 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008792 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008797 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008801 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008806 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008810 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008814 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008818 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008822 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008826 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008830 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008834 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008838 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008847 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008851 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008855 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008860 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008865 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008869 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:34:41.015414 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008873 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008877 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008882 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008886 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008904 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008909 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008912 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008917 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008921 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008925 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008930 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008933 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008936 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008938 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008941 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008944 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008946 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008949 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008952 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:34:41.016266 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008962 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008965 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008968 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008971 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008974 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008976 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008979 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008982 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008984 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008987 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008990 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008992 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008995 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.008998 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009001 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009004 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009007 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009009 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009012 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009015 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:34:41.017078 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009017 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009020 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009022 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009025 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009028 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009030 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009033 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009035 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009038 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009041 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009044 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009046 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009049 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009058 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009061 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009063 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009066 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009068 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009072 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:34:41.017832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009076 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009080 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009083 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009086 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009089 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009092 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009095 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009097 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009100 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009103 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009105 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009108 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009111 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009113 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009117 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009120 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:34:41.018355 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009122 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.009164 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009205 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009209 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009212 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009215 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009218 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009220 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009223 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009226 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009228 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009237 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009240 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009243 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009245 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009248 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:34:41.019055 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009251 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009254 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009256 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009259 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009262 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009265 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009267 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009271 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009273 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009276 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009278 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009281 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009283 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009286 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009288 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009291 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009293 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009296 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009298 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009301 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:34:41.019741 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009303 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009306 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009309 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009311 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009314 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009316 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009318 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009321 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009329 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009332 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009334 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009338 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009340 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009343 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009345 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009347 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009350 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009352 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009355 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:34:41.020584 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009357 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009360 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009363 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009365 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009368 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009370 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009373 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009375 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009378 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009380 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009383 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009385 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009388 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009390 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009393 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009395 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009398 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009401 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009403 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009406 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:34:41.021182 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009408 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009411 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009421 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009425 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009428 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009431 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009434 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009437 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009440 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009442 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009444 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009447 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009450 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009452 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009455 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009459 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009462 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009464 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009467 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:34:41.021852 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009470 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009472 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009474 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009477 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009480 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009482 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009485 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009487 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009490 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009492 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009495 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009500 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009503 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.009506 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.017642 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.017662 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.018323 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.018335 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018448 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:34:41.022689 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018457 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018462 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018475 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018480 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018487 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018493 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018498 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018504 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018508 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018513 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018517 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018522 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018526 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018530 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018535 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018540 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018544 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018549 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018553 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:34:41.023249 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018557 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018561 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018566 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018570 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018574 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018579 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018583 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018587 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018592 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018597 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018604 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018609 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018613 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018619 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018623 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018627 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018632 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018636 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018641 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:34:41.023865 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018645 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018649 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018653 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018658 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018662 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018666 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018671 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018675 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018679 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018684 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018688 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018692 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018697 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018701 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018705 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018710 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018714 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018718 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018723 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018727 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:34:41.024382 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018732 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018736 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018741 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018746 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018751 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018756 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018760 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018764 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018768 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018773 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018777 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018781 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018785 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018790 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018794 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018799 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018803 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018808 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018812 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018816 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:34:41.024958 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018820 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018825 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018829 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018833 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018837 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018841 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018846 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018850 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018854 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018861 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018868 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018873 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018878 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018883 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018903 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018908 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018912 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018918 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018922 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:34:41.025497 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018926 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018931 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018935 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018939 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018943 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018948 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018953 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.018957 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.019003 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019257 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019268 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019275 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019280 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019284 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019289 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019293 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:34:41.025976 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019297 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019302 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019306 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019311 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019315 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019319 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019323 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019328 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019333 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019337 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019342 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019346 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019351 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019356 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019360 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019366 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019370 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019375 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019379 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:34:41.026362 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019383 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019388 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019403 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019409 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019414 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019418 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019423 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019427 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019431 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019436 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019440 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019444 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019448 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019453 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019457 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019461 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019466 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019470 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019475 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:34:41.026825 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019479 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019483 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019488 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019492 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019496 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019500 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019505 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019509 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019514 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019518 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019523 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019528 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019532 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019536 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019540 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019545 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019549 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019554 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019558 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019562 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:34:41.027350 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019566 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019570 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019574 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019578 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019583 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019587 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019592 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019596 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019601 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019605 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019609 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019613 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019618 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019622 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019626 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019631 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019635 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019639 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019644 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019648 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:34:41.027828 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019653 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019657 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019661 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019666 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019673 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019679 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019683 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019688 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019693 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019697 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019701 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019706 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019710 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019715 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019719 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019723 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019728 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019732 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019736 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019740 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:34:41.028325 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019745 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.019787 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019842 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019849 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019855 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019861 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019866 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019870 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019875 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019879 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019884 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019903 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019908 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019912 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019917 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019921 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:34:41.028792 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019926 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019930 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019935 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019939 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019943 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019948 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019952 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019957 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019961 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019965 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019970 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019974 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019979 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019983 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019988 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019992 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.019996 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020001 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020005 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:34:41.029196 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020009 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020019 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020026 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020031 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020035 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020040 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020044 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020048 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020053 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020057 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020062 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020066 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020070 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020075 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020085 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020089 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020094 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020098 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020102 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:34:41.029652 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020107 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020111 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020116 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020120 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020124 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020129 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020134 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020138 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020142 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020146 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020150 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020154 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020159 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020163 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020168 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020172 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020177 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020181 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020185 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020190 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:34:41.030147 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020194 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020198 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020202 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020207 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020211 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020216 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020220 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020224 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020229 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020233 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020237 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020241 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020245 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020250 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020254 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020259 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020263 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020267 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020271 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020275 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:34:41.030646 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020279 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020288 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020293 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020300 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020307 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020311 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020317 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020322 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020326 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020331 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020335 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020340 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020344 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:41.020348 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.020433 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.023596 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.024851 2581 server.go:1014] "Starting client certificate rotation" Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.024971 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 09:34:41.031155 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.025008 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 09:34:41.031599 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.027269 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 09:34:41.031599 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.028518 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 09:34:41.041041 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.041022 2581 log.go:25] "Validated CRI v1 runtime API" Nov 02 09:34:41.046810 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.046787 2581 log.go:25] "Validated CRI v1 image API" Nov 02 09:34:41.047606 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.047579 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 09:34:41.052200 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.052175 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 97f9c2fe-476b-4347-af4a-2180e928b025:/dev/nvme0n1p3 e5d73044-bfe5-40d9-a32d-92b43fe42ae7:/dev/nvme0n1p4] Nov 02 09:34:41.052271 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.052198 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 09:34:41.058378 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.058254 2581 manager.go:217] Machine: {Timestamp:2026-11-02 09:34:41.056770421 +0000 UTC m=+0.341700367 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098558 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e383783e2893636ee295ad01c4bec SystemUUID:ec2e3837-83e2-8936-36ee-295ad01c4bec BootID:53e63310-996b-479f-a3e8-2b690d0f9b97 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:27:c9:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:27:c9:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:02:16:e5:85:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 09:34:41.058378 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.058374 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 09:34:41.058513 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.058458 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 09:34:41.059872 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.059855 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 09:34:41.061503 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.061471 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 09:34:41.061650 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.061506 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-234.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 09:34:41.061696 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.061652 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 09:34:41.061696 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.061660 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 09:34:41.061746 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.061698 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 09:34:41.061746 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.061719 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 09:34:41.063034 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.063023 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 09:34:41.067592 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.067579 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 09:34:41.067725 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.067715 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 09:34:41.067755 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.067750 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 02 09:34:41.067782 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.067762 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 09:34:41.067782 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.067773 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 09:34:41.067782 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.067782 2581 kubelet.go:405] "Adding apiserver pod source" Nov 02 09:34:41.067905 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.067791 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 09:34:41.068154 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.068142 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 09:34:41.070839 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.070821 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 09:34:41.078406 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.078388 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 09:34:41.079411 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.079398 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 09:34:41.079499 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.079430 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 09:34:41.079593 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.079580 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 09:34:41.079629 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.079596 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 09:34:41.079629 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.079603 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 09:34:41.079629 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.079608 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 09:34:41.079629 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.079614 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 09:34:41.079629 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.079620 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 09:34:41.080430 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.080421 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 09:34:41.080466 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.080431 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 09:34:41.080466 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.080440 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 09:34:41.080466 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.080446 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 09:34:41.080466 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.080455 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 09:34:41.080466 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.080464 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 09:34:41.081349 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.081328 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 09:34:41.081349 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.081341 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 09:34:41.085667 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.085649 2581 server.go:1263] "Started kubelet" Nov 02 09:34:41.085783 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.085683 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 09:34:41.085848 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.085781 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 09:34:41.085913 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.085876 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 09:34:41.086453 ip-10-0-138-234 systemd[1]: Started Kubernetes Kubelet. Nov 02 09:34:41.087085 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.086959 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-234.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 09:34:41.087085 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.086967 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 09:34:41.087085 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.087031 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-234.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 09:34:41.087310 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.087288 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 09:34:41.089740 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.089719 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 02 09:34:41.089848 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.089756 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v5rtg" Nov 02 09:34:41.092572 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.092529 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 09:34:41.095327 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.095307 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v5rtg" Nov 02 09:34:41.095490 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.095471 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 09:34:41.095641 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.094547 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-234.us-east-2.compute.internal.18e430af4df1f153 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-234.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-234.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 09:34:41.085411667 +0000 UTC m=+0.370341612,LastTimestamp:2026-11-02 09:34:41.085411667 +0000 UTC m=+0.370341612,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-234.us-east-2.compute.internal,}" Nov 02 09:34:41.096079 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.096059 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 09:34:41.096204 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.096184 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.096740 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.096722 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 09:34:41.096829 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.096810 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 09:34:41.096829 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.096826 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 09:34:41.096988 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.096970 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 02 09:34:41.097041 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.096989 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 02 09:34:41.097105 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.097085 2581 factory.go:55] Registering systemd factory Nov 02 09:34:41.097105 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.097104 2581 factory.go:223] Registration of the systemd container factory successfully Nov 02 09:34:41.097429 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.097414 2581 factory.go:153] Registering CRI-O factory Nov 02 09:34:41.097480 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.097433 2581 factory.go:223] Registration of the crio container factory successfully Nov 02 09:34:41.097520 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.097487 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 09:34:41.097520 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.097505 2581 factory.go:103] Registering Raw factory Nov 02 09:34:41.097594 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.097520 2581 manager.go:1199] Started watching for new ooms in manager Nov 02 09:34:41.098649 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.098624 2581 manager.go:319] Starting recovery of all containers Nov 02 09:34:41.099731 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.099709 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-234.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 09:34:41.109473 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.109454 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:34:41.110974 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.110958 2581 manager.go:324] Recovery completed Nov 02 09:34:41.115677 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.115662 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 09:34:41.117821 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.117804 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 09:34:41.117905 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.117837 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 09:34:41.117905 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.117849 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 09:34:41.118432 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.118416 2581 cpu_manager.go:226] "Starting" policy="none" Nov 02 09:34:41.118432 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.118431 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 09:34:41.118580 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.118449 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 09:34:41.118580 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.118423 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 09:34:41.119341 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.119319 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 09:34:41.119428 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.119372 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 09:34:41.119428 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.119399 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 09:34:41.120467 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.120450 2581 policy_none.go:50] "Start" Nov 02 09:34:41.120501 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.120468 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 09:34:41.120501 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.120479 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 09:34:41.122410 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.122399 2581 policy_none.go:44] "Start" Nov 02 09:34:41.173663 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.173625 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 09:34:41.176236 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.176220 2581 manager.go:341] "Starting Device Plugin manager" Nov 02 09:34:41.176340 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.176259 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 09:34:41.176340 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.176270 2581 server.go:87] "Starting device plugin registration server" Nov 02 09:34:41.176565 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.176551 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 09:34:41.176614 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.176566 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 09:34:41.176700 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.176683 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 09:34:41.176792 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.176765 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 09:34:41.176792 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.176792 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 09:34:41.177307 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.177290 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 09:34:41.177382 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.177334 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.236616 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.236445 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 09:34:41.238621 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.237990 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 09:34:41.238621 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.238021 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 09:34:41.238621 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.238045 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 09:34:41.238621 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.238079 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 09:34:41.242181 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.242161 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:34:41.277332 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.277263 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 09:34:41.278297 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.278277 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 09:34:41.278402 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.278309 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 09:34:41.278402 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.278323 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 09:34:41.278402 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.278355 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.287043 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.287027 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.287109 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.287052 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-234.us-east-2.compute.internal\": node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.306451 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.306426 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.338245 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.338211 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal"] Nov 02 09:34:41.338376 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.338288 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 09:34:41.339432 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.339413 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 09:34:41.339527 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.339445 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 09:34:41.339527 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.339456 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 09:34:41.341733 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.341718 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 09:34:41.341907 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.341860 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.341975 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.341917 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 09:34:41.342615 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.342598 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 09:34:41.342615 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.342604 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 09:34:41.342755 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.342626 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 09:34:41.342755 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.342630 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 09:34:41.342755 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.342636 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 09:34:41.342755 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.342650 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 09:34:41.344819 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.344801 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.344922 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.344829 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 09:34:41.345727 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.345709 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 09:34:41.345826 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.345745 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 09:34:41.345826 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.345759 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 09:34:41.367348 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.367326 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" node="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.370127 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.370108 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" node="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.398915 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.398873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.399048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.398920 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.399048 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.398939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba5c7edc9657ad596c668f03e09721a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7ba5c7edc9657ad596c668f03e09721a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.407014 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.406990 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.499120 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.499079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.499290 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.499116 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.499290 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.499157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba5c7edc9657ad596c668f03e09721a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7ba5c7edc9657ad596c668f03e09721a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.499290 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.499202 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba5c7edc9657ad596c668f03e09721a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7ba5c7edc9657ad596c668f03e09721a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.499290 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.499206 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.499290 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.499206 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.507475 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.507449 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.607610 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.607579 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.671711 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.671684 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.673950 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:41.673927 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:41.708386 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.708351 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.809302 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.809261 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:41.909827 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:41.909754 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:42.010473 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:42.010432 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:42.024993 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.024970 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 09:34:42.025135 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.025115 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 09:34:42.025205 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.025155 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 09:34:42.102055 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.101997 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 09:29:41 +0000 UTC" deadline="2028-04-05 11:56:41.246973455 +0000 UTC" Nov 02 09:34:42.102055 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.102055 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12482h21m59.144932382s" Nov 02 09:34:42.111235 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:42.111216 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-234.us-east-2.compute.internal\" not found" Nov 02 09:34:42.163425 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.163347 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:34:42.196950 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.196918 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:42.211525 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.211503 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 09:34:42.213454 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.213427 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:34:42.223492 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.223473 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 09:34:42.240476 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.240447 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:34:42.322905 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:42.322870 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dde2da7a6b8a26ba7501a7689619da1.slice/crio-620c54e56ffcf8e00930ba7dadbbfb83d4b409a164e8b019234cc0de3dee7fba WatchSource:0}: Error finding container 620c54e56ffcf8e00930ba7dadbbfb83d4b409a164e8b019234cc0de3dee7fba: Status 404 returned error can't find the container with id 620c54e56ffcf8e00930ba7dadbbfb83d4b409a164e8b019234cc0de3dee7fba Nov 02 09:34:42.329735 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:42.329713 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 09:34:42.332810 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:42.332786 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba5c7edc9657ad596c668f03e09721a.slice/crio-4cdde665f3f19c8c14fb425a853a6da18f2cec5d1e6b714a5baf1ea478240a26 WatchSource:0}: Error finding container 4cdde665f3f19c8c14fb425a853a6da18f2cec5d1e6b714a5baf1ea478240a26: Status 404 returned error can't find the container with id 4cdde665f3f19c8c14fb425a853a6da18f2cec5d1e6b714a5baf1ea478240a26 Nov 02 09:34:43.069210 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.069180 2581 apiserver.go:52] "Watching apiserver" Nov 02 09:34:43.078586 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.078558 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 09:34:43.089300 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.089256 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-4xcxp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-pvfk5","openshift-network-diagnostics/network-check-target-j4f9g","openshift-ovn-kubernetes/ovnkube-node-9n577","kube-system/konnectivity-agent-8t5nd","openshift-cluster-node-tuning-operator/tuned-7tfw7","openshift-multus/multus-f52r7","kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal","openshift-image-registry/node-ca-6kvjb","openshift-multus/multus-additional-cni-plugins-v8f95","openshift-network-operator/iptables-alerter-lhvv8","kube-system/global-pull-secret-syncer-zxrg9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq"] Nov 02 09:34:43.091943 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.091916 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.095154 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.095132 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.095271 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.095228 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:43.095743 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.095720 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 09:34:43.096418 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.096396 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:43.096531 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.096477 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:43.096531 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.096514 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-982vl\"" Nov 02 09:34:43.096632 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.096609 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 09:34:43.096688 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.096655 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 09:34:43.099093 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.099073 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:43.099200 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.099141 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:43.101207 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.101185 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.104916 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.104098 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:43.106510 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.106086 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107059 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrkzz\" (UniqueName: \"kubernetes.io/projected/6d1d3a9e-5aab-4313-9626-e0f7772fe862-kube-api-access-xrkzz\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107138 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-dbus\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-lib-modules\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-var-lib-kubelet\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107368 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysconfig\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107392 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-kubernetes\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-systemd\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-run\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-sys\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107525 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-conf\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-host\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-tuned\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.108881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-tmp\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.109683 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107664 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvkc\" (UniqueName: \"kubernetes.io/projected/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-kube-api-access-9fvkc\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.109683 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-kubelet-config\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.109683 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-modprobe-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.109683 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.107775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.109683 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.108355 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.110802 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.110782 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.113621 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.113602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.114238 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.114209 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 09:34:43.117595 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.116186 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.118698 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.118675 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.132111 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.132088 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9dn6w" Nov 02 09:34:43.140019 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.139997 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9dn6w" Nov 02 09:34:43.197155 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.197121 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.197841 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.197906 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.197937 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.197841 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.197987 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.197994 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198052 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198056 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 09:34:43.198094 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198060 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 09:34:43.198538 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198235 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 09:34:43.198596 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198583 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2j22\"" Nov 02 09:34:43.198642 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198627 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 09:34:43.198691 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198642 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 09:34:43.198741 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198701 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 09:34:43.198741 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198720 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 09:34:43.198837 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198795 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 09:34:43.198885 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.198809 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 09:34:43.199408 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.199385 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbp4s\"" Nov 02 09:34:43.199507 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.199424 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 09:34:43.199507 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.199468 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 09:34:43.199507 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.199495 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 09:34:43.200072 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200045 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-llzhk\"" Nov 02 09:34:43.200181 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200118 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t9b8s\"" Nov 02 09:34:43.200311 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200298 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 09:34:43.200401 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200380 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 09:34:43.200509 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200468 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 09:34:43.200509 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200502 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7vqld\"" Nov 02 09:34:43.200605 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200583 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 09:34:43.200637 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200623 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 09:34:43.200675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200645 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 09:34:43.200745 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200730 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9tqmn\"" Nov 02 09:34:43.200827 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200801 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zc5hs\"" Nov 02 09:34:43.200866 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.200843 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kdt4x\"" Nov 02 09:34:43.208246 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208215 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-socket-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.208388 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c7eba7-d115-4796-89a5-47a4b7af0789-host\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.208388 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv85g\" (UniqueName: \"kubernetes.io/projected/2f0d6935-5f2d-488d-9441-3dd6c418b832-kube-api-access-cv85g\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.208388 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208310 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-etc-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.208388 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-multus-daemon-config\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.208388 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.208388 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208368 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-config\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-kubernetes\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-sys\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-kubelet-config\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208448 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82bk\" (UniqueName: \"kubernetes.io/projected/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-kube-api-access-l82bk\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-host\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208496 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-tmp\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvkc\" (UniqueName: \"kubernetes.io/projected/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-kube-api-access-9fvkc\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208530 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-systemd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-socket-dir-parent\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.208585 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-netns\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-multus\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208608 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-conf-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-system-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208644 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-multus-certs\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-dbus\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208682 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-cnibin\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208708 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f0d6935-5f2d-488d-9441-3dd6c418b832-iptables-alerter-script\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7qh\" (UniqueName: \"kubernetes.io/projected/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kube-api-access-2f7qh\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208821 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.208886 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.208835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-kubelet\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.209410 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209248 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-dbus\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.209410 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209271 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-sys\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.209410 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209308 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-kubernetes\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.209410 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-systemd\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.209523 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-netns\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.209523 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209420 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-host\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.209523 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-registration-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.209523 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-sys-fs\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.209636 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209523 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-systemd\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.209636 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06c7eba7-d115-4796-89a5-47a4b7af0789-serviceca\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.209636 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209593 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-kubelet-config\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.209636 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209590 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 09:34:43.209796 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209626 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-tuned\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.209796 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209701 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-var-lib-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.209796 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-ovn\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.209796 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc719dc6-3e71-4f85-b7fc-51cdee604e74-agent-certs\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:43.209970 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209815 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-tmp-dir\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.209970 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-modprobe-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.209970 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.209970 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.209900 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.210152 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.209990 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:43.210152 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210010 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.210152 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkzz\" (UniqueName: \"kubernetes.io/projected/6d1d3a9e-5aab-4313-9626-e0f7772fe862-kube-api-access-xrkzz\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:43.210152 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210083 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-modprobe-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.210152 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.210115 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret podName:568d527d-7b27-4e41-a946-1893e04d2653 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:43.71008769 +0000 UTC m=+2.995017640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret") pod "global-pull-secret-syncer-zxrg9" (UID: "568d527d-7b27-4e41-a946-1893e04d2653") : object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:43.210363 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210148 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.210363 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-log-socket\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.210363 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-bin\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.210363 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210342 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovn-node-metrics-cert\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.210502 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-systemd-units\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.210502 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-node-log\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.210502 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210435 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.210502 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgk8\" (UniqueName: \"kubernetes.io/projected/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-kube-api-access-gjgk8\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.210670 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc719dc6-3e71-4f85-b7fc-51cdee604e74-konnectivity-ca\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:43.210670 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210581 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.210670 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210643 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-cnibin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.210806 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-lib-modules\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.210806 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-var-lib-kubelet\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.210806 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysconfig\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.210806 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210793 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-lib-modules\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.211024 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-slash\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.211024 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210838 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-var-lib-kubelet\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.211024 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-device-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.211024 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210880 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysconfig\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.211024 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-cni-binary-copy\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.211024 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.210968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-k8s-cni-cncf-io\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.211234 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-bin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.211234 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.211234 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-conf\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.211234 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5cz\" (UniqueName: \"kubernetes.io/projected/9db316d5-bf73-458b-ba54-9e4126840510-kube-api-access-6v5cz\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.211234 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211183 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-etc-kubernetes\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.211234 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:43.211430 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211240 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:43.211430 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.211430 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d798b\" (UniqueName: \"kubernetes.io/projected/06c7eba7-d115-4796-89a5-47a4b7af0789-kube-api-access-d798b\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.211430 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-run\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.211430 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-kubelet\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.211430 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-env-overrides\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.211675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-system-cni-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.211675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0d6935-5f2d-488d-9441-3dd6c418b832-host-slash\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.211675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-netd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.211675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211526 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-hosts-file\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.211675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-hostroot\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.211675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211572 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktbp\" (UniqueName: \"kubernetes.io/projected/2a880a57-b5db-4f8f-944f-741952727830-kube-api-access-9ktbp\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.211675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211599 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.211675 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211624 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-script-lib\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.212057 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.211801 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:43.212057 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211810 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-conf\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.212057 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.211869 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:43.711851308 +0000 UTC m=+2.996781266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:43.212057 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.211942 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-run\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.213591 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.213566 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-tuned\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.214347 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.214318 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-tmp\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.229771 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.229742 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvkc\" (UniqueName: \"kubernetes.io/projected/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-kube-api-access-9fvkc\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.231272 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.231238 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkzz\" (UniqueName: \"kubernetes.io/projected/6d1d3a9e-5aab-4313-9626-e0f7772fe862-kube-api-access-xrkzz\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:43.231710 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.231689 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 09:34:43.231824 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.231714 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 09:34:43.231824 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.231727 2581 projected.go:196] Error preparing data for projected volume kube-api-access-fn55t for pod openshift-network-diagnostics/network-check-target-j4f9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:43.231824 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.231797 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t podName:3267a968-7d5c-4392-a775-65f20929c5bc nodeName:}" failed. No retries permitted until 2026-11-02 09:34:43.731777811 +0000 UTC m=+3.016707763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fn55t" (UniqueName: "kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t") pod "network-check-target-j4f9g" (UID: "3267a968-7d5c-4392-a775-65f20929c5bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:43.243725 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.243574 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7ba5c7edc9657ad596c668f03e09721a","Type":"ContainerStarted","Data":"4cdde665f3f19c8c14fb425a853a6da18f2cec5d1e6b714a5baf1ea478240a26"} Nov 02 09:34:43.245814 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.245755 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7dde2da7a6b8a26ba7501a7689619da1","Type":"ContainerStarted","Data":"620c54e56ffcf8e00930ba7dadbbfb83d4b409a164e8b019234cc0de3dee7fba"} Nov 02 09:34:43.312426 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312367 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-log-socket\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312426 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-bin\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312638 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312479 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-bin\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312638 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312490 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-log-socket\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312638 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovn-node-metrics-cert\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312638 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-systemd-units\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312638 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-node-log\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312638 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312638 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgk8\" (UniqueName: \"kubernetes.io/projected/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-kube-api-access-gjgk8\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312635 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-systemd-units\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc719dc6-3e71-4f85-b7fc-51cdee604e74-konnectivity-ca\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312686 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-cnibin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-slash\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-device-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-cni-binary-copy\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-k8s-cni-cncf-io\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-bin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.312904 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5cz\" (UniqueName: \"kubernetes.io/projected/9db316d5-bf73-458b-ba54-9e4126840510-kube-api-access-6v5cz\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-etc-kubernetes\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d798b\" (UniqueName: \"kubernetes.io/projected/06c7eba7-d115-4796-89a5-47a4b7af0789-kube-api-access-d798b\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-kubelet\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-env-overrides\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-system-cni-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0d6935-5f2d-488d-9441-3dd6c418b832-host-slash\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-netd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-hosts-file\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-hostroot\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktbp\" (UniqueName: \"kubernetes.io/projected/2a880a57-b5db-4f8f-944f-741952727830-kube-api-access-9ktbp\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313311 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc719dc6-3e71-4f85-b7fc-51cdee604e74-konnectivity-ca\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313312 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.313364 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-script-lib\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-socket-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c7eba7-d115-4796-89a5-47a4b7af0789-host\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv85g\" (UniqueName: \"kubernetes.io/projected/2f0d6935-5f2d-488d-9441-3dd6c418b832-kube-api-access-cv85g\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-etc-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-multus-daemon-config\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313483 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-config\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82bk\" (UniqueName: \"kubernetes.io/projected/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-kube-api-access-l82bk\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-systemd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313590 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-socket-dir-parent\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-netns\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-multus\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-conf-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-system-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-multus-certs\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313707 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.314123 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-cnibin\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313760 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f0d6935-5f2d-488d-9441-3dd6c418b832-iptables-alerter-script\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7qh\" (UniqueName: \"kubernetes.io/projected/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kube-api-access-2f7qh\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-kubelet\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313835 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-netns\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313860 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313927 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-cnibin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.312729 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-node-log\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313955 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-registration-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313991 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-sys-fs\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.313991 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-etc-kubernetes\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06c7eba7-d115-4796-89a5-47a4b7af0789-serviceca\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314019 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-device-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.314927 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-var-lib-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-ovn\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314064 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314120 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc719dc6-3e71-4f85-b7fc-51cdee604e74-agent-certs\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314136 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-kubelet\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-tmp-dir\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314607 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-cni-binary-copy\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314660 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-env-overrides\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314676 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-k8s-cni-cncf-io\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314728 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-system-cni-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314725 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-bin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314779 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0d6935-5f2d-488d-9441-3dd6c418b832-host-slash\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314802 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-hosts-file\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314853 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-netd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314913 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-hostroot\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.314907 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315107 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.315710 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315210 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-sys-fs\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315534 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06c7eba7-d115-4796-89a5-47a4b7af0789-serviceca\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315593 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315768 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-var-lib-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315803 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315810 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-script-lib\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315821 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-ovn\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315830 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-kubelet\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315871 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-netns\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315870 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-systemd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315930 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-slash\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c7eba7-d115-4796-89a5-47a4b7af0789-host\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315963 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-conf-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.315994 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-netns\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316020 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-registration-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316035 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316027 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-multus\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316047 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-socket-dir-parent\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316461 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316066 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-etc-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.316976 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316101 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-multus-certs\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316976 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316147 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-system-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316976 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316215 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-cnibin\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.316976 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316394 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-socket-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.316976 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316715 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-multus-daemon-config\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.316976 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316733 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f0d6935-5f2d-488d-9441-3dd6c418b832-iptables-alerter-script\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.316976 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.316922 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovn-node-metrics-cert\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.317252 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.317001 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-tmp-dir\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.317252 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.317165 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-config\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.319120 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.319088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc719dc6-3e71-4f85-b7fc-51cdee604e74-agent-certs\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:43.336579 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.336465 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82bk\" (UniqueName: \"kubernetes.io/projected/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-kube-api-access-l82bk\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.337287 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.337255 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv85g\" (UniqueName: \"kubernetes.io/projected/2f0d6935-5f2d-488d-9441-3dd6c418b832-kube-api-access-cv85g\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.340218 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.340192 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d798b\" (UniqueName: \"kubernetes.io/projected/06c7eba7-d115-4796-89a5-47a4b7af0789-kube-api-access-d798b\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.340674 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.340652 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7qh\" (UniqueName: \"kubernetes.io/projected/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kube-api-access-2f7qh\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.344881 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.344469 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5cz\" (UniqueName: \"kubernetes.io/projected/9db316d5-bf73-458b-ba54-9e4126840510-kube-api-access-6v5cz\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.346064 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.346038 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktbp\" (UniqueName: \"kubernetes.io/projected/2a880a57-b5db-4f8f-944f-741952727830-kube-api-access-9ktbp\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.347046 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.347026 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgk8\" (UniqueName: \"kubernetes.io/projected/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-kube-api-access-gjgk8\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.412445 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.412406 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:34:43.423721 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.423693 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7faf99_47b5_4aa1_8af8_6d4ba29fa6d1.slice/crio-b459443cd072d5bade65d7f556e397553600c09cc47e514356358e6aa44db439 WatchSource:0}: Error finding container b459443cd072d5bade65d7f556e397553600c09cc47e514356358e6aa44db439: Status 404 returned error can't find the container with id b459443cd072d5bade65d7f556e397553600c09cc47e514356358e6aa44db439 Nov 02 09:34:43.435477 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.435449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:43.438439 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.438409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:34:43.444263 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.444231 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:34:43.447327 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.447301 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:34:43.448358 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.448330 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc719dc6_3e71_4f85_b7fc_51cdee604e74.slice/crio-c0b2dec5ab7f3602e42fd8dc5cd559dce30835814caced894672f06ddf91d5b9 WatchSource:0}: Error finding container c0b2dec5ab7f3602e42fd8dc5cd559dce30835814caced894672f06ddf91d5b9: Status 404 returned error can't find the container with id c0b2dec5ab7f3602e42fd8dc5cd559dce30835814caced894672f06ddf91d5b9 Nov 02 09:34:43.448747 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.448722 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f265c2b_4beb_4f7b_bed5_d91ef6c00377.slice/crio-36c64add1ba65ba53cdfe82684b9e7a5fb2fc714414a7259b107861de0b19faa WatchSource:0}: Error finding container 36c64add1ba65ba53cdfe82684b9e7a5fb2fc714414a7259b107861de0b19faa: Status 404 returned error can't find the container with id 36c64add1ba65ba53cdfe82684b9e7a5fb2fc714414a7259b107861de0b19faa Nov 02 09:34:43.453195 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.453174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f52r7" Nov 02 09:34:43.456485 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.456453 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be0dcba_1629_4bc9_9cc2_77e736cdbac3.slice/crio-41abe3a5e051d7f7cc31768290069bffbd09626962cb71427bd23b5a3937cf8d WatchSource:0}: Error finding container 41abe3a5e051d7f7cc31768290069bffbd09626962cb71427bd23b5a3937cf8d: Status 404 returned error can't find the container with id 41abe3a5e051d7f7cc31768290069bffbd09626962cb71427bd23b5a3937cf8d Nov 02 09:34:43.457159 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.457135 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad130bd_a2d7_40a9_8d1e_d6a728b7d49e.slice/crio-0a528e75759da5962195b15f2a8af2c13b6a548ac8c474800b5397339e9f657c WatchSource:0}: Error finding container 0a528e75759da5962195b15f2a8af2c13b6a548ac8c474800b5397339e9f657c: Status 404 returned error can't find the container with id 0a528e75759da5962195b15f2a8af2c13b6a548ac8c474800b5397339e9f657c Nov 02 09:34:43.460965 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.460941 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:34:43.462832 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.462810 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a880a57_b5db_4f8f_944f_741952727830.slice/crio-ab514a92ea3dab13ecfda9a6ee08276c96fc67135206921f4ee52ad6dfd2988a WatchSource:0}: Error finding container ab514a92ea3dab13ecfda9a6ee08276c96fc67135206921f4ee52ad6dfd2988a: Status 404 returned error can't find the container with id ab514a92ea3dab13ecfda9a6ee08276c96fc67135206921f4ee52ad6dfd2988a Nov 02 09:34:43.472319 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.472292 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c7eba7_d115_4796_89a5_47a4b7af0789.slice/crio-088a0edf852154cb50fd6b4212ed5500960c521610f290192bcc8cc46c133f28 WatchSource:0}: Error finding container 088a0edf852154cb50fd6b4212ed5500960c521610f290192bcc8cc46c133f28: Status 404 returned error can't find the container with id 088a0edf852154cb50fd6b4212ed5500960c521610f290192bcc8cc46c133f28 Nov 02 09:34:43.481143 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.481112 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:34:43.482530 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.482503 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:34:43.494836 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.494807 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db316d5_bf73_458b_ba54_9e4126840510.slice/crio-f3aa6bfc2b798908fbfab594de184bf21b3cc4598706a28bfea98faebe663534 WatchSource:0}: Error finding container f3aa6bfc2b798908fbfab594de184bf21b3cc4598706a28bfea98faebe663534: Status 404 returned error can't find the container with id f3aa6bfc2b798908fbfab594de184bf21b3cc4598706a28bfea98faebe663534 Nov 02 09:34:43.498608 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:34:43.497992 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0d6935_5f2d_488d_9441_3dd6c418b832.slice/crio-183c6057c5038a7ae1a13f1dec45085349cbeb33a7894d27237f18b974967e94 WatchSource:0}: Error finding container 183c6057c5038a7ae1a13f1dec45085349cbeb33a7894d27237f18b974967e94: Status 404 returned error can't find the container with id 183c6057c5038a7ae1a13f1dec45085349cbeb33a7894d27237f18b974967e94 Nov 02 09:34:43.499660 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.499612 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:34:43.615337 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.615254 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:34:43.717327 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.717279 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:43.717498 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.717352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:43.717498 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.717433 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:43.717498 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.717452 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:43.717498 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.717499 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:44.717481222 +0000 UTC m=+4.002411171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:43.717698 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.717516 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret podName:568d527d-7b27-4e41-a946-1893e04d2653 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:44.717507476 +0000 UTC m=+4.002437411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret") pod "global-pull-secret-syncer-zxrg9" (UID: "568d527d-7b27-4e41-a946-1893e04d2653") : object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:43.818662 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:43.818633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:43.818830 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.818787 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 09:34:43.818830 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.818803 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 09:34:43.818830 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.818813 2581 projected.go:196] Error preparing data for projected volume kube-api-access-fn55t for pod openshift-network-diagnostics/network-check-target-j4f9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:43.819016 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:43.818861 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t podName:3267a968-7d5c-4392-a775-65f20929c5bc nodeName:}" failed. No retries permitted until 2026-11-02 09:34:44.818849321 +0000 UTC m=+4.103779258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn55t" (UniqueName: "kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t") pod "network-check-target-j4f9g" (UID: "3267a968-7d5c-4392-a775-65f20929c5bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:44.143786 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.143640 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:43 +0000 UTC" deadline="2028-05-11 15:42:50.618584196 +0000 UTC" Nov 02 09:34:44.143786 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.143776 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13350h8m6.474812751s" Nov 02 09:34:44.274302 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.274097 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" event={"ID":"6be0dcba-1629-4bc9-9cc2-77e736cdbac3","Type":"ContainerStarted","Data":"41abe3a5e051d7f7cc31768290069bffbd09626962cb71427bd23b5a3937cf8d"} Nov 02 09:34:44.285968 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.285835 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"36c64add1ba65ba53cdfe82684b9e7a5fb2fc714414a7259b107861de0b19faa"} Nov 02 09:34:44.299580 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.299545 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8t5nd" event={"ID":"dc719dc6-3e71-4f85-b7fc-51cdee604e74","Type":"ContainerStarted","Data":"c0b2dec5ab7f3602e42fd8dc5cd559dce30835814caced894672f06ddf91d5b9"} Nov 02 09:34:44.304389 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.304340 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" event={"ID":"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1","Type":"ContainerStarted","Data":"b459443cd072d5bade65d7f556e397553600c09cc47e514356358e6aa44db439"} Nov 02 09:34:44.314083 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.314010 2581 generic.go:356] "Generic (PLEG): container finished" podID="7dde2da7a6b8a26ba7501a7689619da1" containerID="9a38d4e29fbb7e1dc959d67e8d5e20123a92788ace7f891d79013b02ebf628db" exitCode=0 Nov 02 09:34:44.314269 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.314106 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7dde2da7a6b8a26ba7501a7689619da1","Type":"ContainerDied","Data":"9a38d4e29fbb7e1dc959d67e8d5e20123a92788ace7f891d79013b02ebf628db"} Nov 02 09:34:44.315611 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.315565 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lhvv8" event={"ID":"2f0d6935-5f2d-488d-9441-3dd6c418b832","Type":"ContainerStarted","Data":"183c6057c5038a7ae1a13f1dec45085349cbeb33a7894d27237f18b974967e94"} Nov 02 09:34:44.317590 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.317565 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8f95" event={"ID":"9db316d5-bf73-458b-ba54-9e4126840510","Type":"ContainerStarted","Data":"f3aa6bfc2b798908fbfab594de184bf21b3cc4598706a28bfea98faebe663534"} Nov 02 09:34:44.321729 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.321701 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6kvjb" event={"ID":"06c7eba7-d115-4796-89a5-47a4b7af0789","Type":"ContainerStarted","Data":"088a0edf852154cb50fd6b4212ed5500960c521610f290192bcc8cc46c133f28"} Nov 02 09:34:44.326282 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.326256 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f52r7" event={"ID":"2a880a57-b5db-4f8f-944f-741952727830","Type":"ContainerStarted","Data":"ab514a92ea3dab13ecfda9a6ee08276c96fc67135206921f4ee52ad6dfd2988a"} Nov 02 09:34:44.327911 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.327875 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4xcxp" event={"ID":"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e","Type":"ContainerStarted","Data":"0a528e75759da5962195b15f2a8af2c13b6a548ac8c474800b5397339e9f657c"} Nov 02 09:34:44.725138 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.724862 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:44.725375 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:44.725151 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:44.725375 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.725200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:44.725375 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:44.725228 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret podName:568d527d-7b27-4e41-a946-1893e04d2653 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.725207585 +0000 UTC m=+6.010137534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret") pod "global-pull-secret-syncer-zxrg9" (UID: "568d527d-7b27-4e41-a946-1893e04d2653") : object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:44.725375 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:44.725349 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:44.725645 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:44.725412 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.725395937 +0000 UTC m=+6.010325893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:44.826492 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:44.826450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:44.826744 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:44.826644 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 09:34:44.826744 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:44.826665 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 09:34:44.826744 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:44.826678 2581 projected.go:196] Error preparing data for projected volume kube-api-access-fn55t for pod openshift-network-diagnostics/network-check-target-j4f9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:44.826957 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:44.826745 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t podName:3267a968-7d5c-4392-a775-65f20929c5bc nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.826727573 +0000 UTC m=+6.111657512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn55t" (UniqueName: "kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t") pod "network-check-target-j4f9g" (UID: "3267a968-7d5c-4392-a775-65f20929c5bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:45.144564 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:45.144474 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:43 +0000 UTC" deadline="2028-07-23 11:01:16.603352544 +0000 UTC" Nov 02 09:34:45.144564 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:45.144516 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15097h26m31.458841466s" Nov 02 09:34:45.250183 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:45.250152 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:45.250357 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:45.250320 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:45.250915 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:45.250825 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:45.251072 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:45.250991 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:45.252752 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:45.251579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:45.252752 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:45.251728 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:45.338721 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:45.338058 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7ba5c7edc9657ad596c668f03e09721a","Type":"ContainerStarted","Data":"380ae39c5072f1d6d67326182093cbf2f04ed9dca5b19dd04f0fae338f7d11ec"} Nov 02 09:34:45.345663 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:45.345149 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7dde2da7a6b8a26ba7501a7689619da1","Type":"ContainerStarted","Data":"532a5a1ce23baba89e054e4dcda74908e291ee7c48c0762cb91ca809d3725218"} Nov 02 09:34:45.372598 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:45.372537 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" podStartSLOduration=3.372518397 podStartE2EDuration="3.372518397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:45.354799902 +0000 UTC m=+4.639729859" watchObservedRunningTime="2026-11-02 09:34:45.372518397 +0000 UTC m=+4.657448350" Nov 02 09:34:46.747970 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:46.747874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:46.748429 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:46.748002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:46.748429 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:46.748152 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:46.748429 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:46.748236 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret podName:568d527d-7b27-4e41-a946-1893e04d2653 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:50.748205604 +0000 UTC m=+10.033135543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret") pod "global-pull-secret-syncer-zxrg9" (UID: "568d527d-7b27-4e41-a946-1893e04d2653") : object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:46.748685 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:46.748664 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:46.748755 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:46.748725 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:50.74870757 +0000 UTC m=+10.033637558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:46.848900 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:46.848846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:46.849113 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:46.849095 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 09:34:46.849217 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:46.849125 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 09:34:46.849217 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:46.849139 2581 projected.go:196] Error preparing data for projected volume kube-api-access-fn55t for pod openshift-network-diagnostics/network-check-target-j4f9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:46.849217 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:46.849198 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t podName:3267a968-7d5c-4392-a775-65f20929c5bc nodeName:}" failed. No retries permitted until 2026-11-02 09:34:50.849180279 +0000 UTC m=+10.134110219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn55t" (UniqueName: "kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t") pod "network-check-target-j4f9g" (UID: "3267a968-7d5c-4392-a775-65f20929c5bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:47.239794 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:47.239404 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:47.239794 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:47.239404 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:47.239794 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:47.239552 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:47.239794 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:47.239650 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:47.239794 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:47.239703 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:47.239794 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:47.239759 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:47.366471 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:47.366434 2581 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="873cc7b5b241e06c3c80318ff73214b05d3f88de024280035630ebcf2dc86404" exitCode=0 Nov 02 09:34:47.366670 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:47.366544 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8f95" event={"ID":"9db316d5-bf73-458b-ba54-9e4126840510","Type":"ContainerDied","Data":"873cc7b5b241e06c3c80318ff73214b05d3f88de024280035630ebcf2dc86404"} Nov 02 09:34:47.395810 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:47.394824 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" podStartSLOduration=5.394810637 podStartE2EDuration="5.394810637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:45.373923113 +0000 UTC m=+4.658853068" watchObservedRunningTime="2026-11-02 09:34:47.394810637 +0000 UTC m=+6.679740591" Nov 02 09:34:48.385218 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:48.385175 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8t5nd" event={"ID":"dc719dc6-3e71-4f85-b7fc-51cdee604e74","Type":"ContainerStarted","Data":"8c75ca4a1bca2a4e463da040c40d5b00de96e068386f6553b7409953dfaa6948"} Nov 02 09:34:49.238956 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.238919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:49.239125 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:49.239076 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:49.239552 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.239530 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:49.239649 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:49.239629 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:49.239851 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.239831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:49.240054 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:49.239955 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:49.391410 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.391374 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6kvjb" event={"ID":"06c7eba7-d115-4796-89a5-47a4b7af0789","Type":"ContainerStarted","Data":"2034d6c50bb77285b9278f1c1ee8d8ee9490e46ca22ac49d3508a05ceb5d41a2"} Nov 02 09:34:49.400975 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.400943 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" event={"ID":"6be0dcba-1629-4bc9-9cc2-77e736cdbac3","Type":"ContainerStarted","Data":"aaa0b578525699df124520b691a63a09145f503f274e7db99999e9b3936929f1"} Nov 02 09:34:49.407588 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.407547 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8t5nd" podStartSLOduration=4.659937001 podStartE2EDuration="8.407528298s" totalImagesPullingTime="3.747591297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:48.404977094 +0000 UTC m=+7.689907050" watchObservedRunningTime="2026-11-02 09:34:49.407528298 +0000 UTC m=+8.692458255" Nov 02 09:34:49.974241 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.974172 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:49.974844 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.974819 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:34:49.992064 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:49.992005 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6kvjb" podStartSLOduration=3.61866413 podStartE2EDuration="8.991986061s" totalImagesPullingTime="5.373321931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:49.408599765 +0000 UTC m=+8.693529719" watchObservedRunningTime="2026-11-02 09:34:49.991986061 +0000 UTC m=+9.276916017" Nov 02 09:34:50.785130 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:50.784582 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:50.785130 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:50.784627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:50.785130 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:50.784751 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:50.785130 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:50.784799 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:58.784786755 +0000 UTC m=+18.069716700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:50.785130 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:50.785012 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:50.785130 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:50.785059 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret podName:568d527d-7b27-4e41-a946-1893e04d2653 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:58.785044677 +0000 UTC m=+18.069974615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret") pod "global-pull-secret-syncer-zxrg9" (UID: "568d527d-7b27-4e41-a946-1893e04d2653") : object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:50.885703 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:50.885670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:50.885878 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:50.885810 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 09:34:50.885878 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:50.885826 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 09:34:50.885878 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:50.885835 2581 projected.go:196] Error preparing data for projected volume kube-api-access-fn55t for pod openshift-network-diagnostics/network-check-target-j4f9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:50.885878 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:50.885878 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t podName:3267a968-7d5c-4392-a775-65f20929c5bc nodeName:}" failed. No retries permitted until 2026-11-02 09:34:58.885865833 +0000 UTC m=+18.170795778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn55t" (UniqueName: "kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t") pod "network-check-target-j4f9g" (UID: "3267a968-7d5c-4392-a775-65f20929c5bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:51.208106 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:51.208060 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 09:34:51.242406 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:51.241931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:51.242406 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:51.241957 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:51.242406 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:51.241936 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:51.242406 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:51.242034 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:51.242677 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:51.242467 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:51.242677 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:51.242546 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:51.408651 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:51.408571 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f52r7" event={"ID":"2a880a57-b5db-4f8f-944f-741952727830","Type":"ContainerStarted","Data":"cca805b1adcd82e5f3dedba2ed2fbf2b6d312f88de01f545a6dfcb85ff188b7c"} Nov 02 09:34:51.421916 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:51.421866 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 09:34:51.422313 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:51.422291 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" event={"ID":"6be0dcba-1629-4bc9-9cc2-77e736cdbac3","Type":"ContainerStarted","Data":"5535dcea3358443e6fecc96fede76d6c7f1d9b10c6c6d3648c5365ce078b0d2b"} Nov 02 09:34:52.186543 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.186417 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T09:34:51.208079718Z","UUID":"d79702ec-f518-430a-b00d-b452bc3783fb","Handler":null,"Name":"","Endpoint":""} Nov 02 09:34:52.188698 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.188289 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 09:34:52.188698 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.188320 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 09:34:52.426226 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.425788 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lhvv8" event={"ID":"2f0d6935-5f2d-488d-9441-3dd6c418b832","Type":"ContainerStarted","Data":"637b7c369c59fcb73b908d492b5604522af22bcc35afb817639076289a178a52"} Nov 02 09:34:52.428558 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.428529 2581 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="87ae2379d36f616fa363d9b922f42bc9cda5d109c898cf56f80ec263772a939d" exitCode=0 Nov 02 09:34:52.428686 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.428625 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8f95" event={"ID":"9db316d5-bf73-458b-ba54-9e4126840510","Type":"ContainerDied","Data":"87ae2379d36f616fa363d9b922f42bc9cda5d109c898cf56f80ec263772a939d"} Nov 02 09:34:52.431342 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.431246 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4xcxp" event={"ID":"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e","Type":"ContainerStarted","Data":"089098d4d8988117468bb9a94dc9337ced1fe3bfe58c0745f2f9eaefac16eab7"} Nov 02 09:34:52.443314 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.442335 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-f52r7" podStartSLOduration=3.939037067 podStartE2EDuration="11.442313668s" totalImagesPullingTime="7.503276601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:51.432496481 +0000 UTC m=+10.717426438" watchObservedRunningTime="2026-11-02 09:34:52.442313668 +0000 UTC m=+11.727243626" Nov 02 09:34:52.443314 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:52.442681 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lhvv8" podStartSLOduration=3.813454344 podStartE2EDuration="11.442666219s" totalImagesPullingTime="7.629211875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:52.442283675 +0000 UTC m=+11.727213631" watchObservedRunningTime="2026-11-02 09:34:52.442666219 +0000 UTC m=+11.727596175" Nov 02 09:34:53.238479 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:53.238443 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:53.238928 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:53.238494 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:53.238928 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:53.238441 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:53.238928 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:53.238583 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:53.238928 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:53.238672 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:53.238928 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:53.238752 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:53.436650 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:53.436615 2581 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="f2ae6a3f095d12965708c92af4eb0c264b048f3dd5353cd49291a8c15cc0cbe8" exitCode=0 Nov 02 09:34:53.436797 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:53.436693 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8f95" event={"ID":"9db316d5-bf73-458b-ba54-9e4126840510","Type":"ContainerDied","Data":"f2ae6a3f095d12965708c92af4eb0c264b048f3dd5353cd49291a8c15cc0cbe8"} Nov 02 09:34:53.440234 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:53.440202 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" event={"ID":"6be0dcba-1629-4bc9-9cc2-77e736cdbac3","Type":"ContainerStarted","Data":"63540a6d1c0cdf58fe553e1dc0a2d03b0b700592d5c96db1678da5f2afd199dd"} Nov 02 09:34:53.463201 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:53.463156 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4xcxp" podStartSLOduration=4.422855644 podStartE2EDuration="12.463140678s" totalImagesPullingTime="8.040285034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:52.484292088 +0000 UTC m=+11.769222057" watchObservedRunningTime="2026-11-02 09:34:53.463140678 +0000 UTC m=+12.748070633" Nov 02 09:34:53.480641 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:53.480584 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" podStartSLOduration=4.011324274 podStartE2EDuration="12.480565779s" totalImagesPullingTime="8.469241505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:53.480224642 +0000 UTC m=+12.765154597" watchObservedRunningTime="2026-11-02 09:34:53.480565779 +0000 UTC m=+12.765495735" Nov 02 09:34:54.444401 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:54.444359 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" event={"ID":"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1","Type":"ContainerStarted","Data":"866c29a708741bb8d0dab328f5d48893712f19a6a43bada81fced46e6ebcc74d"} Nov 02 09:34:54.464331 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:54.464286 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" podStartSLOduration=3.495039428 podStartE2EDuration="13.464271931s" totalImagesPullingTime="9.969232503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:54.463810548 +0000 UTC m=+13.748740502" watchObservedRunningTime="2026-11-02 09:34:54.464271931 +0000 UTC m=+13.749201885" Nov 02 09:34:55.239232 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:55.239193 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:55.239398 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:55.239193 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:55.239398 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:55.239331 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:55.239522 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:55.239428 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:55.239522 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:55.239206 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:55.239603 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:55.239557 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:55.449833 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:55.449795 2581 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="f801ccb40abda1d067d48772e8358bfe3f42f9d749dd0215cca312263f8d9575" exitCode=0 Nov 02 09:34:55.450486 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:55.449875 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8f95" event={"ID":"9db316d5-bf73-458b-ba54-9e4126840510","Type":"ContainerDied","Data":"f801ccb40abda1d067d48772e8358bfe3f42f9d749dd0215cca312263f8d9575"} Nov 02 09:34:57.239245 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:57.239015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:57.239623 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:57.239028 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:57.239623 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:57.239355 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:57.239623 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:57.239418 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:57.239623 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:57.239085 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:57.239623 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:57.239495 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:58.459366 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.459156 2581 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="0984e82e421729407267e32819438a92bbff4836b8dfd149f85bdd63ffc37d39" exitCode=0 Nov 02 09:34:58.459783 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.459230 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8f95" event={"ID":"9db316d5-bf73-458b-ba54-9e4126840510","Type":"ContainerDied","Data":"0984e82e421729407267e32819438a92bbff4836b8dfd149f85bdd63ffc37d39"} Nov 02 09:34:58.465239 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.465215 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"9ad95b6489345c18a1a8410f451145e5cb9f6aab554b3f62582cc6f61a0a5218"} Nov 02 09:34:58.465334 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.465305 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"a16a6b2cd8c70ae774df6e8e563ad840a3394b244cdde74777d98d6c45255979"} Nov 02 09:34:58.465368 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.465335 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"981961e0d9cf74a0fa5b7665e944724dfe7ad11d6c0225ab3676df4dc386287d"} Nov 02 09:34:58.465397 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.465367 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"4fb958a6d03be4cb535866f09ece1699942c5af70ad860f60e9c5540e9c2fe31"} Nov 02 09:34:58.465425 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.465396 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"62bf5c70f728b310502ac93f9eb18ff2c3bd596886b0d44bc37b95888cc9a33d"} Nov 02 09:34:58.465425 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.465421 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"1ea31c646b49e358c988fd05e176eab15de0c2f7c696202e9c46e2cbe2ec70ae"} Nov 02 09:34:58.853660 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.853624 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:58.853836 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.853672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:58.853836 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:58.853788 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:58.853836 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:58.853835 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:14.853822354 +0000 UTC m=+34.138752287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 09:34:58.853978 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:58.853788 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:58.853978 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:58.853933 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret podName:568d527d-7b27-4e41-a946-1893e04d2653 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:14.853920388 +0000 UTC m=+34.138850320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret") pod "global-pull-secret-syncer-zxrg9" (UID: "568d527d-7b27-4e41-a946-1893e04d2653") : object "kube-system"/"original-pull-secret" not registered Nov 02 09:34:58.954128 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:58.954093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:58.954275 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:58.954216 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 09:34:58.954275 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:58.954232 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 09:34:58.954275 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:58.954241 2581 projected.go:196] Error preparing data for projected volume kube-api-access-fn55t for pod openshift-network-diagnostics/network-check-target-j4f9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:58.954368 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:58.954291 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t podName:3267a968-7d5c-4392-a775-65f20929c5bc nodeName:}" failed. No retries permitted until 2026-11-02 09:35:14.954278875 +0000 UTC m=+34.239208809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fn55t" (UniqueName: "kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t") pod "network-check-target-j4f9g" (UID: "3267a968-7d5c-4392-a775-65f20929c5bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 09:34:59.238680 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:59.238602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:34:59.238680 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:59.238628 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:34:59.238850 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:59.238700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:34:59.238850 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:59.238756 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:34:59.238935 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:59.238883 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:34:59.238935 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:34:59.238885 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:34:59.470241 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:59.470212 2581 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="ba40d53444995891b189ed5b8193fdd5e4a0efd8af32bd802a7aefc3847b7afb" exitCode=0 Nov 02 09:34:59.470672 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:34:59.470255 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8f95" event={"ID":"9db316d5-bf73-458b-ba54-9e4126840510","Type":"ContainerDied","Data":"ba40d53444995891b189ed5b8193fdd5e4a0efd8af32bd802a7aefc3847b7afb"} Nov 02 09:35:00.476250 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:00.476215 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8f95" event={"ID":"9db316d5-bf73-458b-ba54-9e4126840510","Type":"ContainerStarted","Data":"5fcd6ba909f4b86df7691b3446ddc5bd473524b0f8e62c3ddd333dac2912280d"} Nov 02 09:35:00.479610 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:00.479586 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"1e13f3950f0d35c79d7b02343fe60ffe816c46cf8d5dde16ade9b77f11d4dfe7"} Nov 02 09:35:00.504225 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:00.504170 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v8f95" podStartSLOduration=8.371629999 podStartE2EDuration="19.504154077s" totalImagesPullingTime="11.132524078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:00.503321127 +0000 UTC m=+19.788251083" watchObservedRunningTime="2026-11-02 09:35:00.504154077 +0000 UTC m=+19.789084036" Nov 02 09:35:01.239622 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:01.239593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:01.239757 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:01.239722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:01.239757 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:01.239729 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:35:01.239844 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:01.239748 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:01.239844 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:01.239812 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:35:01.239933 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:01.239915 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:35:03.239383 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:03.239224 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:03.240070 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:03.239224 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:03.240070 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:03.239449 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:35:03.240070 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:03.239252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:03.240070 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:03.239522 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:35:03.240070 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:03.239635 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:35:03.493220 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:03.493144 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" event={"ID":"2f265c2b-4beb-4f7b-bed5-d91ef6c00377","Type":"ContainerStarted","Data":"b4cd7e0452709ecee9c439d9c1e076c406a277fc36646656f4e07441680ecf7a"} Nov 02 09:35:03.493411 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:03.493390 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:03.493466 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:03.493460 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:03.508650 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:03.508628 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:03.525336 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:03.525292 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" podStartSLOduration=7.951145116 podStartE2EDuration="22.525280956s" totalImagesPullingTime="14.57413584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:03.524833757 +0000 UTC m=+22.809763723" watchObservedRunningTime="2026-11-02 09:35:03.525280956 +0000 UTC m=+22.810210953" Nov 02 09:35:04.239372 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:04.239299 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zxrg9"] Nov 02 09:35:04.239506 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:04.239431 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:04.239795 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:04.239587 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:35:04.242603 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:04.242575 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j4f9g"] Nov 02 09:35:04.242715 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:04.242684 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:04.242799 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:04.242778 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:35:04.243325 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:04.243303 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pvfk5"] Nov 02 09:35:04.243432 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:04.243418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:04.243546 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:04.243524 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:35:04.495494 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:04.495411 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:04.511472 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:04.511447 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:06.239025 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:06.238994 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:06.239479 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:06.239037 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:06.239479 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:06.238994 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:06.239479 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:06.239112 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:35:06.239479 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:06.239196 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:35:06.239479 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:06.239251 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:35:08.238438 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:08.238403 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:08.239026 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:08.238403 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:08.239026 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:08.238527 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:35:08.239026 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:08.238403 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:08.239026 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:08.238598 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:35:08.239026 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:08.238659 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:35:10.238611 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.238444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:10.239067 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.238444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:10.239067 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:10.238686 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pvfk5" podUID="6d1d3a9e-5aab-4313-9626-e0f7772fe862" Nov 02 09:35:10.239067 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.238447 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:10.239067 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:10.238758 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxrg9" podUID="568d527d-7b27-4e41-a946-1893e04d2653" Nov 02 09:35:10.239067 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:10.238840 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4f9g" podUID="3267a968-7d5c-4392-a775-65f20929c5bc" Nov 02 09:35:10.433288 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.433253 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeReady" Nov 02 09:35:10.433454 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.433439 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 09:35:10.480376 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.480340 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fffwx"] Nov 02 09:35:10.482879 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.482851 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.483262 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.483109 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8hd89"] Nov 02 09:35:10.485902 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.485874 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:10.486516 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.486500 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 09:35:10.486718 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.486699 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zwj6d\"" Nov 02 09:35:10.487078 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.487062 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 09:35:10.494263 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.494244 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fffwx"] Nov 02 09:35:10.498806 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.498773 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8hd89"] Nov 02 09:35:10.587407 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.587368 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 09:35:10.587407 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.587385 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 09:35:10.587407 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.587402 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gztjj\"" Nov 02 09:35:10.587407 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.587393 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 09:35:10.638224 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.638187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c677-2d64-4973-a7e5-9ba0c04cc22d-tmp-dir\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.638384 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.638236 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/565aea70-fb40-497b-9a8a-876abb6edb02-cert\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:10.638384 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.638306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.638384 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.638331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8790c677-2d64-4973-a7e5-9ba0c04cc22d-config-volume\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.638384 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.638351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgq25\" (UniqueName: \"kubernetes.io/projected/8790c677-2d64-4973-a7e5-9ba0c04cc22d-kube-api-access-tgq25\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.638511 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.638477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhk4\" (UniqueName: \"kubernetes.io/projected/565aea70-fb40-497b-9a8a-876abb6edb02-kube-api-access-8vhk4\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:10.739145 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.739101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/565aea70-fb40-497b-9a8a-876abb6edb02-cert\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:10.739145 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.739151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.739393 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.739180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8790c677-2d64-4973-a7e5-9ba0c04cc22d-config-volume\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.739393 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.739204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq25\" (UniqueName: \"kubernetes.io/projected/8790c677-2d64-4973-a7e5-9ba0c04cc22d-kube-api-access-tgq25\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.739393 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:10.739270 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:10.739393 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:10.739354 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:11.239337288 +0000 UTC m=+30.524267245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:10.739393 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.739387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhk4\" (UniqueName: \"kubernetes.io/projected/565aea70-fb40-497b-9a8a-876abb6edb02-kube-api-access-8vhk4\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:10.739620 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.739424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c677-2d64-4973-a7e5-9ba0c04cc22d-tmp-dir\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.739765 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.739725 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c677-2d64-4973-a7e5-9ba0c04cc22d-tmp-dir\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.739923 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.739882 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8790c677-2d64-4973-a7e5-9ba0c04cc22d-config-volume\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.743391 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.743367 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/565aea70-fb40-497b-9a8a-876abb6edb02-cert\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:10.751821 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.751794 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq25\" (UniqueName: \"kubernetes.io/projected/8790c677-2d64-4973-a7e5-9ba0c04cc22d-kube-api-access-tgq25\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:10.753008 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.752988 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhk4\" (UniqueName: \"kubernetes.io/projected/565aea70-fb40-497b-9a8a-876abb6edb02-kube-api-access-8vhk4\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:10.787075 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.787040 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:10.806563 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:10.806522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:11.017642 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:11.017558 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:11.100077 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:11.100043 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8hd89"] Nov 02 09:35:11.104270 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:35:11.104238 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565aea70_fb40_497b_9a8a_876abb6edb02.slice/crio-27b25240bd9a57bd88c4a821cb6619f53c1e58c50c6ce6714ebe46fbefe4e860 WatchSource:0}: Error finding container 27b25240bd9a57bd88c4a821cb6619f53c1e58c50c6ce6714ebe46fbefe4e860: Status 404 returned error can't find the container with id 27b25240bd9a57bd88c4a821cb6619f53c1e58c50c6ce6714ebe46fbefe4e860 Nov 02 09:35:11.233268 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:11.233235 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:35:11.233429 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:11.233352 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 09:35:11.233816 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:11.233796 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:35:11.244079 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:11.244058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:11.244403 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:11.244187 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:11.244403 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:11.244244 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.244230221 +0000 UTC m=+31.529160166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:11.418026 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:11.417996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:11.513361 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:11.513327 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8hd89" event={"ID":"565aea70-fb40-497b-9a8a-876abb6edb02","Type":"ContainerStarted","Data":"27b25240bd9a57bd88c4a821cb6619f53c1e58c50c6ce6714ebe46fbefe4e860"} Nov 02 09:35:12.238969 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.238927 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:12.238969 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.238958 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:12.239209 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.238977 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:12.243225 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.243205 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 09:35:12.244471 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.244451 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 09:35:12.244764 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.244467 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 09:35:12.244764 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.244451 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9d4cl\"" Nov 02 09:35:12.244764 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.244451 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbhf4\"" Nov 02 09:35:12.244764 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.244473 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sbddj\"" Nov 02 09:35:12.244764 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.244525 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 09:35:12.250870 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.250851 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:12.250994 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:12.250981 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:12.251053 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:12.251043 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:14.251029682 +0000 UTC m=+33.535959629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:12.721953 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.721899 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:12.904951 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:12.904920 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:14.012502 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.012465 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:14.212139 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.212061 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:14.267858 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.267823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:14.268045 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:14.267981 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:14.268093 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:14.268056 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:18.268041712 +0000 UTC m=+37.552971665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:14.872155 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.872115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:14.872346 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.872212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:14.872346 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:14.872279 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 09:35:14.872464 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:14.872349 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:46.872334651 +0000 UTC m=+66.157264600 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : secret "metrics-daemon-secret" not found Nov 02 09:35:14.874665 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.874640 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:14.968236 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.968171 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:14.973483 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.973456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:14.976157 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:14.976136 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:15.021985 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:15.021948 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:15.285466 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:15.285398 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zxrg9"] Nov 02 09:35:15.288286 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:35:15.288264 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568d527d_7b27_4e41_a946_1893e04d2653.slice/crio-72181ef10532878e4a5b4bea37d80745fab7748fa38dbd564b0e4794c99c0424 WatchSource:0}: Error finding container 72181ef10532878e4a5b4bea37d80745fab7748fa38dbd564b0e4794c99c0424: Status 404 returned error can't find the container with id 72181ef10532878e4a5b4bea37d80745fab7748fa38dbd564b0e4794c99c0424 Nov 02 09:35:15.322650 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:15.322609 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j4f9g"] Nov 02 09:35:15.326291 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:35:15.326265 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3267a968_7d5c_4392_a775_65f20929c5bc.slice/crio-1ca635d5e0a4a93b3d5a9852a9cd69a495e962cbec4b0dfda7bc73a03e9a3147 WatchSource:0}: Error finding container 1ca635d5e0a4a93b3d5a9852a9cd69a495e962cbec4b0dfda7bc73a03e9a3147: Status 404 returned error can't find the container with id 1ca635d5e0a4a93b3d5a9852a9cd69a495e962cbec4b0dfda7bc73a03e9a3147 Nov 02 09:35:15.523977 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:15.523940 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j4f9g" event={"ID":"3267a968-7d5c-4392-a775-65f20929c5bc","Type":"ContainerStarted","Data":"1ca635d5e0a4a93b3d5a9852a9cd69a495e962cbec4b0dfda7bc73a03e9a3147"} Nov 02 09:35:15.524906 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:15.524870 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zxrg9" event={"ID":"568d527d-7b27-4e41-a946-1893e04d2653","Type":"ContainerStarted","Data":"72181ef10532878e4a5b4bea37d80745fab7748fa38dbd564b0e4794c99c0424"} Nov 02 09:35:16.432170 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.431141 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cs7kk"] Nov 02 09:35:16.453864 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.453833 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cs7kk"] Nov 02 09:35:16.454040 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.453988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.549738 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.549699 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:16.551926 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.551572 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8hd89" event={"ID":"565aea70-fb40-497b-9a8a-876abb6edb02","Type":"ContainerStarted","Data":"06df1d4c678f5de77ff55ffef6c6a2297c4f4f7282b95ee0bb89f2d5daf5beb8"} Nov 02 09:35:16.554531 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.554506 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 09:35:16.554639 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.554578 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 09:35:16.554714 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.554699 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 09:35:16.555210 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.555173 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 09:35:16.555363 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.555345 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrsh4\"" Nov 02 09:35:16.570309 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.570248 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8hd89" podStartSLOduration=2.115250169 podStartE2EDuration="6.570234001s" totalImagesPullingTime="4.454983832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:16.569542022 +0000 UTC m=+35.854471979" watchObservedRunningTime="2026-11-02 09:35:16.570234001 +0000 UTC m=+35.855163955" Nov 02 09:35:16.584072 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.584039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6mc\" (UniqueName: \"kubernetes.io/projected/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-api-access-vt6mc\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.584231 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.584082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-crio-socket\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.584231 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.584110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.584231 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.584126 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.584231 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.584159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-data-volume\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.685342 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.685246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6mc\" (UniqueName: \"kubernetes.io/projected/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-api-access-vt6mc\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.685342 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.685304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-crio-socket\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.685556 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.685350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.685556 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.685376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.685556 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.685412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-data-volume\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.686419 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:16.685849 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 09:35:16.686419 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:16.685973 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:17.185952571 +0000 UTC m=+36.470882504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : configmap "kube-rbac-proxy" not found Nov 02 09:35:16.686419 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.686121 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-data-volume\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.686419 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:16.686232 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 09:35:16.686419 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:16.686284 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:17.186266858 +0000 UTC m=+36.471196804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : secret "insights-runtime-extractor-tls" not found Nov 02 09:35:16.686733 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.686427 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-crio-socket\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.703832 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.703803 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6mc\" (UniqueName: \"kubernetes.io/projected/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-api-access-vt6mc\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:16.757365 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.757323 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:16.816946 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.816908 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m"] Nov 02 09:35:16.837420 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.837368 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m"] Nov 02 09:35:16.837570 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.837486 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" Nov 02 09:35:16.844491 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.844460 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6hxzv\"" Nov 02 09:35:16.848724 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.848695 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 09:35:16.848724 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.848713 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 09:35:16.936961 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.936858 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:16.968564 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.968527 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2wf4v"] Nov 02 09:35:16.983517 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.983488 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:16.984523 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.984498 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2wf4v"] Nov 02 09:35:16.987496 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.987475 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 09:35:16.987590 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.987560 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 09:35:16.987657 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.987631 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pzrq9\"" Nov 02 09:35:16.987709 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.987678 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 09:35:16.987709 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.987680 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 09:35:16.988028 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:16.988003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ph8j\" (UniqueName: \"kubernetes.io/projected/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc-kube-api-access-7ph8j\") pod \"migrator-66df5d496b-b792m\" (UID: \"8dcc0e95-455b-4a11-bb79-ff4e9d3338dc\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" Nov 02 09:35:17.089369 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.089329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8w89\" (UniqueName: \"kubernetes.io/projected/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-kube-api-access-c8w89\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.089553 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.089407 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-key\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.089617 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.089553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ph8j\" (UniqueName: \"kubernetes.io/projected/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc-kube-api-access-7ph8j\") pod \"migrator-66df5d496b-b792m\" (UID: \"8dcc0e95-455b-4a11-bb79-ff4e9d3338dc\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" Nov 02 09:35:17.089617 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.089606 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-cabundle\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.102272 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.102246 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ph8j\" (UniqueName: \"kubernetes.io/projected/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc-kube-api-access-7ph8j\") pod \"migrator-66df5d496b-b792m\" (UID: \"8dcc0e95-455b-4a11-bb79-ff4e9d3338dc\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" Nov 02 09:35:17.113285 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.113254 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:17.164294 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.164237 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" Nov 02 09:35:17.190535 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.190502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:17.190649 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.190547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:17.190649 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.190587 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-cabundle\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.190649 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.190622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8w89\" (UniqueName: \"kubernetes.io/projected/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-kube-api-access-c8w89\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.190978 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.190681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-key\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.191990 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:17.191134 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 09:35:17.191990 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:17.191204 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:18.191183805 +0000 UTC m=+37.476113743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : secret "insights-runtime-extractor-tls" not found Nov 02 09:35:17.191990 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:17.191511 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 09:35:17.191990 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:17.191546 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:18.191535693 +0000 UTC m=+37.476465640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : configmap "kube-rbac-proxy" not found Nov 02 09:35:17.191990 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.191810 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-cabundle\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.195008 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.194981 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-key\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.205193 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.205055 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8w89\" (UniqueName: \"kubernetes.io/projected/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-kube-api-access-c8w89\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.311816 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.311778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:17.397252 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.397215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:17.465463 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.465377 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m"] Nov 02 09:35:17.470618 ip-10-0-138-234 kubenswrapper[2581]: W1102 09:35:17.470568 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcc0e95_455b_4a11_bb79_ff4e9d3338dc.slice/crio-21c65e7a8199528b63b2e795f25216edebf3e1a1097c8d3b97d8dea92c82616d WatchSource:0}: Error finding container 21c65e7a8199528b63b2e795f25216edebf3e1a1097c8d3b97d8dea92c82616d: Status 404 returned error can't find the container with id 21c65e7a8199528b63b2e795f25216edebf3e1a1097c8d3b97d8dea92c82616d Nov 02 09:35:17.556069 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.556030 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" event={"ID":"8dcc0e95-455b-4a11-bb79-ff4e9d3338dc","Type":"ContainerStarted","Data":"21c65e7a8199528b63b2e795f25216edebf3e1a1097c8d3b97d8dea92c82616d"} Nov 02 09:35:17.574920 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.574874 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:17.610074 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:17.610037 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2wf4v"] Nov 02 09:35:18.200206 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:18.200103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:18.200206 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:18.200149 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:18.200414 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:18.200247 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 09:35:18.200414 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:18.200293 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 09:35:18.200414 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:18.200327 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:20.200306661 +0000 UTC m=+39.485236595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : configmap "kube-rbac-proxy" not found Nov 02 09:35:18.200414 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:18.200361 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:20.200343224 +0000 UTC m=+39.485273160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : secret "insights-runtime-extractor-tls" not found Nov 02 09:35:18.301168 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:18.301133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:18.301360 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:18.301270 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:18.301360 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:18.301319 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:26.30130628 +0000 UTC m=+45.586236217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:18.335949 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:18.335905 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:18.560678 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:18.560569 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" event={"ID":"5a71ee3d-860c-4201-9be3-8fc7e7404cbe","Type":"ContainerStarted","Data":"c954f2e92759af3ce9b778caf0d383dfde8d2fdd22329f064e5f4c6abd93482e"} Nov 02 09:35:18.793489 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:18.793452 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:19.567239 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:19.567199 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j4f9g" event={"ID":"3267a968-7d5c-4392-a775-65f20929c5bc","Type":"ContainerStarted","Data":"a091a5f87378c32f70e549afb29d51042fb828d11cf13aaa63881ab4c8bb5864"} Nov 02 09:35:19.567991 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:19.567959 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:19.570011 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:19.569984 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" event={"ID":"8dcc0e95-455b-4a11-bb79-ff4e9d3338dc","Type":"ContainerStarted","Data":"cb784444aa2f641c40b2fc6b749f564f5e3243ae90e651c82512dd06dfa1a920"} Nov 02 09:35:19.585315 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:19.585262 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j4f9g" podStartSLOduration=34.468624278 podStartE2EDuration="38.585250338s" totalImagesPullingTime="4.11662606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:19.584580452 +0000 UTC m=+38.869510408" watchObservedRunningTime="2026-11-02 09:35:19.585250338 +0000 UTC m=+38.870180287" Nov 02 09:35:19.601478 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:19.601428 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" podStartSLOduration=1.628886202 podStartE2EDuration="3.601411907s" totalImagesPullingTime="1.972525705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:19.600718885 +0000 UTC m=+38.885648840" watchObservedRunningTime="2026-11-02 09:35:19.601411907 +0000 UTC m=+38.886341864" Nov 02 09:35:20.217249 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:20.217202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:20.217433 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:20.217252 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:20.217433 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:20.217422 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 09:35:20.217551 ip-10-0-138-234 kubenswrapper[2581]: E1102 09:35:20.217493 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.217476209 +0000 UTC m=+43.502406167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : secret "insights-runtime-extractor-tls" not found Nov 02 09:35:20.217854 ip-10-0-138-234 kubenswrapper[2581]: I1102 09:35:20.217827 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:21.078707 ip-10-0-138-234 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 09:35:21.098391 ip-10-0-138-234 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 09:35:21.098620 ip-10-0-138-234 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 09:35:21.098719 ip-10-0-138-234 systemd[1]: kubelet.service: Consumed 2.381s CPU time. Nov 02 09:35:21.112589 ip-10-0-138-234 systemd[1]: Starting Kubernetes Kubelet... Nov 02 09:35:21.227990 ip-10-0-138-234 kubenswrapper[4467]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 09:35:21.227990 ip-10-0-138-234 kubenswrapper[4467]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 09:35:21.227990 ip-10-0-138-234 kubenswrapper[4467]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 09:35:21.227990 ip-10-0-138-234 kubenswrapper[4467]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 09:35:21.230537 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230444 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:35:21.230537 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230535 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:35:21.230537 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230541 4467 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:35:21.230537 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230546 4467 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230550 4467 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230555 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230559 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230564 4467 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230568 4467 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230571 4467 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230576 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230580 4467 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230584 4467 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230588 4467 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230591 4467 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230603 4467 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230608 4467 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230612 4467 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230616 4467 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230620 4467 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230624 4467 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230628 4467 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:35:21.230830 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230632 4467 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230636 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230639 4467 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230643 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230648 4467 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230652 4467 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230656 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230660 4467 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230664 4467 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230668 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230672 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230676 4467 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230679 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230684 4467 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230688 4467 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230693 4467 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230698 4467 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230702 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230706 4467 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230711 4467 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:35:21.231723 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230715 4467 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230719 4467 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230724 4467 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230728 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230733 4467 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230738 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230742 4467 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230746 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230750 4467 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230755 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230759 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230763 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230767 4467 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230772 4467 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230776 4467 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230780 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230784 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230789 4467 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230796 4467 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:35:21.232643 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230803 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230809 4467 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230814 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230818 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230822 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230826 4467 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230831 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230835 4467 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230840 4467 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230845 4467 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230849 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230853 4467 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230857 4467 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230862 4467 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230867 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230871 4467 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230877 4467 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230882 4467 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230901 4467 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230906 4467 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:35:21.233490 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230912 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230916 4467 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230920 4467 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230924 4467 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230929 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230933 4467 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230937 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230941 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230952 4467 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230958 4467 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230962 4467 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230966 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230971 4467 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230975 4467 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230979 4467 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230984 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230988 4467 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230993 4467 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.230998 4467 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:35:21.234395 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.231003 4467 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.231011 4467 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.231017 4467 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.231022 4467 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.231028 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.231032 4467 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232297 4467 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232309 4467 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232314 4467 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232318 4467 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232323 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232330 4467 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232334 4467 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232339 4467 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232343 4467 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232347 4467 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232351 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232356 4467 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232360 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232364 4467 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:35:21.235281 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232368 4467 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232372 4467 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232377 4467 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232382 4467 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232386 4467 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232390 4467 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232395 4467 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232399 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232403 4467 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232407 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232411 4467 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232415 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232419 4467 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232423 4467 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232427 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232432 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232436 4467 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232440 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232445 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:35:21.236210 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232449 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232453 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232457 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232461 4467 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232465 4467 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232470 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232475 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232479 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232484 4467 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232488 4467 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232492 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232497 4467 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232503 4467 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232509 4467 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232513 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232519 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232523 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232527 4467 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232531 4467 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:35:21.237029 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232536 4467 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232540 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232544 4467 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232550 4467 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232557 4467 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232562 4467 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232567 4467 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232572 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232577 4467 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232581 4467 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232585 4467 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232591 4467 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232595 4467 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232599 4467 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232603 4467 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232608 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232612 4467 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232616 4467 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232620 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232625 4467 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:35:21.237718 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232630 4467 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232634 4467 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232638 4467 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232643 4467 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232647 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232651 4467 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232655 4467 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232659 4467 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232663 4467 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232668 4467 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232672 4467 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232677 4467 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232681 4467 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232685 4467 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232690 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232694 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232699 4467 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232703 4467 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232708 4467 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:35:21.238451 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232712 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232716 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232720 4467 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232725 4467 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232731 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232735 4467 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232739 4467 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232743 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232747 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232751 4467 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232755 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232759 4467 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232763 4467 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232769 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.232773 4467 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233594 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233603 4467 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233608 4467 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233613 4467 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:35:21.239200 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233620 4467 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233625 4467 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233629 4467 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233633 4467 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233638 4467 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233642 4467 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233646 4467 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233651 4467 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233655 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233660 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233664 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233668 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233672 4467 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233676 4467 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233681 4467 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233685 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233689 4467 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233693 4467 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233698 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233702 4467 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:35:21.239840 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233706 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233710 4467 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233715 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233719 4467 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233723 4467 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233727 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233733 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233737 4467 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233741 4467 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233746 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233750 4467 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233754 4467 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233759 4467 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233763 4467 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233768 4467 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233772 4467 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233777 4467 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233781 4467 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233785 4467 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233790 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:35:21.240574 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233794 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233799 4467 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233803 4467 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233807 4467 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233811 4467 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233815 4467 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233819 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233824 4467 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233828 4467 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233832 4467 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233837 4467 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233841 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233845 4467 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233849 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233859 4467 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233865 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233871 4467 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233875 4467 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233880 4467 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233885 4467 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:35:21.241283 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233907 4467 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233911 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233915 4467 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233919 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233923 4467 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233928 4467 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233932 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233936 4467 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233940 4467 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233944 4467 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233949 4467 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233953 4467 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233957 4467 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233963 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233967 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233971 4467 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233976 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233980 4467 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233984 4467 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:35:21.242044 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233991 4467 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.233997 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234004 4467 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234009 4467 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234013 4467 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234017 4467 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234022 4467 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234029 4467 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234033 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234037 4467 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234042 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234046 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234051 4467 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234056 4467 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234062 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234066 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234071 4467 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234075 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234080 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:35:21.242728 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234084 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234088 4467 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234092 4467 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.234098 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234238 4467 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234249 4467 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234259 4467 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234265 4467 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234272 4467 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234279 4467 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234286 4467 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234292 4467 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234298 4467 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234303 4467 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234308 4467 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234314 4467 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234320 4467 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234324 4467 flags.go:64] FLAG: --cgroup-root="" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234329 4467 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234334 4467 flags.go:64] FLAG: --client-ca-file="" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234340 4467 flags.go:64] FLAG: --cloud-provider="external" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234345 4467 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234351 4467 flags.go:64] FLAG: --cluster-domain="" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234356 4467 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 09:35:21.243470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234362 4467 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234367 4467 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234373 4467 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234380 4467 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234385 4467 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234390 4467 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234396 4467 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234400 4467 flags.go:64] FLAG: --contention-profiling="false" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234405 4467 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234409 4467 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234414 4467 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234421 4467 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234428 4467 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234434 4467 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234438 4467 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234443 4467 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234448 4467 flags.go:64] FLAG: --enable-server="true" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234453 4467 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234460 4467 flags.go:64] FLAG: --event-burst="100" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234465 4467 flags.go:64] FLAG: --event-qps="50" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234470 4467 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234475 4467 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234480 4467 flags.go:64] FLAG: --eviction-hard="" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234487 4467 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 09:35:21.244386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234492 4467 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234497 4467 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234502 4467 flags.go:64] FLAG: --eviction-soft="" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234506 4467 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234513 4467 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234518 4467 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234523 4467 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234527 4467 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234532 4467 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234537 4467 flags.go:64] FLAG: --feature-gates="" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234547 4467 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234552 4467 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234566 4467 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234571 4467 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234576 4467 flags.go:64] FLAG: --healthz-port="10248" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234581 4467 flags.go:64] FLAG: --help="false" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234586 4467 flags.go:64] FLAG: --hostname-override="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234591 4467 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234596 4467 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234603 4467 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234609 4467 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234614 4467 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234619 4467 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 09:35:21.245205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234624 4467 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234629 4467 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234633 4467 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234638 4467 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234643 4467 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234648 4467 flags.go:64] FLAG: --kube-reserved="" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234653 4467 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234658 4467 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234664 4467 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234668 4467 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234673 4467 flags.go:64] FLAG: --lock-file="" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234678 4467 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234682 4467 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234689 4467 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234698 4467 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234703 4467 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234708 4467 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234713 4467 flags.go:64] FLAG: --logging-format="text" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234717 4467 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234723 4467 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234728 4467 flags.go:64] FLAG: --manifest-url="" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234733 4467 flags.go:64] FLAG: --manifest-url-header="" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234747 4467 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234753 4467 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234759 4467 flags.go:64] FLAG: --max-pods="110" Nov 02 09:35:21.246243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234764 4467 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234769 4467 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234774 4467 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234780 4467 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234786 4467 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234791 4467 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234795 4467 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234814 4467 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234819 4467 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234824 4467 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234829 4467 flags.go:64] FLAG: --pod-cidr="" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234834 4467 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234838 4467 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234856 4467 flags.go:64] FLAG: --pods-per-core="0" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234862 4467 flags.go:64] FLAG: --port="10250" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234867 4467 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234872 4467 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01995ce47ff987087" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234877 4467 flags.go:64] FLAG: --qos-reserved="" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234882 4467 flags.go:64] FLAG: --read-only-port="10255" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234902 4467 flags.go:64] FLAG: --register-node="true" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234909 4467 flags.go:64] FLAG: --register-with-taints="" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234915 4467 flags.go:64] FLAG: --registry-burst="10" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234920 4467 flags.go:64] FLAG: --registry-qps="5" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234924 4467 flags.go:64] FLAG: --reserved-cpus="" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234929 4467 flags.go:64] FLAG: --reserved-memory="" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234935 4467 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 09:35:21.247182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234940 4467 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234944 4467 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234956 4467 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234961 4467 flags.go:64] FLAG: --runonce="false" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234974 4467 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234979 4467 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234984 4467 flags.go:64] FLAG: --seccomp-default="false" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234989 4467 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.234994 4467 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235001 4467 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235006 4467 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235011 4467 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235016 4467 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235021 4467 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235025 4467 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235030 4467 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235035 4467 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235040 4467 flags.go:64] FLAG: --system-cgroups="" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235045 4467 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235054 4467 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235058 4467 flags.go:64] FLAG: --tls-cert-file="" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235063 4467 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235069 4467 flags.go:64] FLAG: --tls-min-version="" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235074 4467 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235079 4467 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 09:35:21.248227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235083 4467 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235091 4467 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235096 4467 flags.go:64] FLAG: --v="2" Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235102 4467 flags.go:64] FLAG: --version="false" Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235109 4467 flags.go:64] FLAG: --vmodule="" Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235115 4467 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235120 4467 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235349 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235357 4467 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235362 4467 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235367 4467 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235372 4467 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235377 4467 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235381 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235386 4467 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235390 4467 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235398 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235402 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235407 4467 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235411 4467 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235415 4467 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:35:21.249110 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235420 4467 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235427 4467 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235433 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235438 4467 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235443 4467 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235447 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235452 4467 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235456 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235461 4467 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235465 4467 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235469 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235474 4467 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235480 4467 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235484 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235491 4467 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235496 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235501 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235505 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235510 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:35:21.249641 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235515 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235519 4467 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235523 4467 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235527 4467 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235532 4467 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235537 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235542 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235546 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235553 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235557 4467 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235562 4467 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235566 4467 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235571 4467 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235575 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235579 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235583 4467 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235588 4467 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235592 4467 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235597 4467 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235601 4467 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:35:21.250208 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235605 4467 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235610 4467 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235614 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235618 4467 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235623 4467 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235629 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235634 4467 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235638 4467 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235642 4467 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235647 4467 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235651 4467 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235655 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235659 4467 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235663 4467 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235667 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235672 4467 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235676 4467 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235680 4467 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235686 4467 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235690 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:35:21.251012 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235696 4467 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235700 4467 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235705 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235709 4467 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235713 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235717 4467 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235722 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235726 4467 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235731 4467 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235735 4467 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235739 4467 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235744 4467 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235748 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235752 4467 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235756 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235761 4467 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235765 4467 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235771 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235775 4467 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:35:21.251736 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235780 4467 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235784 4467 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235788 4467 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235793 4467 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235797 4467 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235801 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235805 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235809 4467 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235813 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235818 4467 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235822 4467 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235827 4467 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235832 4467 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235837 4467 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.235878 4467 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235954 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:35:21.252384 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235962 4467 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235967 4467 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235972 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235977 4467 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235982 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235986 4467 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235990 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235994 4467 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.235999 4467 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236004 4467 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236008 4467 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236013 4467 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236017 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236022 4467 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236026 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236032 4467 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236036 4467 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236041 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236045 4467 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:35:21.252834 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236049 4467 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236054 4467 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236058 4467 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236063 4467 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236067 4467 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236071 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236075 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236079 4467 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236084 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236088 4467 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236095 4467 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236099 4467 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236104 4467 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236108 4467 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236112 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236117 4467 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236121 4467 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236126 4467 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236130 4467 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236134 4467 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:35:21.253520 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236139 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236143 4467 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236147 4467 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236152 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236156 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236160 4467 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236164 4467 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236168 4467 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236177 4467 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236184 4467 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236190 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236195 4467 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236201 4467 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236206 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236210 4467 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236214 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236218 4467 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236222 4467 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236226 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:35:21.254290 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236230 4467 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236234 4467 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236239 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236248 4467 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236254 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236259 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236264 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236269 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236274 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236278 4467 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236282 4467 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236287 4467 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236291 4467 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236296 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236300 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236305 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236309 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236313 4467 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236318 4467 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:35:21.255025 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236322 4467 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236326 4467 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236332 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236337 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236341 4467 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236346 4467 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236350 4467 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236354 4467 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236358 4467 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236363 4467 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236367 4467 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236371 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236376 4467 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236381 4467 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236385 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236390 4467 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236395 4467 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236400 4467 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236405 4467 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236410 4467 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:35:21.255727 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236414 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236418 4467 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236422 4467 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236427 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236431 4467 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236435 4467 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236439 4467 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.236445 4467 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.241992 4467 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.242014 4467 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.242036 4467 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.242044 4467 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242116 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242124 4467 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242130 4467 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242134 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242140 4467 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242143 4467 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242146 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:35:21.256474 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242150 4467 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242155 4467 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242160 4467 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242165 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242169 4467 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242173 4467 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242178 4467 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242182 4467 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242186 4467 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242190 4467 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242194 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242199 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242203 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242207 4467 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242211 4467 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242215 4467 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242220 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242225 4467 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242230 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242234 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:35:21.257178 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242239 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242242 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242254 4467 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242259 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242263 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242268 4467 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242272 4467 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242275 4467 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242278 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242280 4467 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242283 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242285 4467 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242288 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242291 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242293 4467 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242296 4467 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242299 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242302 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242306 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242311 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:35:21.257949 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242315 4467 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242320 4467 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242324 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242330 4467 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242334 4467 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242338 4467 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242342 4467 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242351 4467 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242358 4467 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242364 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242370 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242375 4467 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242381 4467 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242387 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242393 4467 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242405 4467 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242410 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242414 4467 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242421 4467 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:35:21.258710 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242428 4467 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242433 4467 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242438 4467 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242442 4467 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242446 4467 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242450 4467 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242454 4467 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242458 4467 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242463 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242467 4467 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242472 4467 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242476 4467 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242480 4467 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242485 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242489 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242494 4467 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242499 4467 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242503 4467 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242507 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:35:21.259404 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242512 4467 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242516 4467 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242521 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242525 4467 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242529 4467 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242533 4467 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242538 4467 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242542 4467 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242546 4467 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242550 4467 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242564 4467 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242569 4467 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242573 4467 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242577 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242581 4467 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242585 4467 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242589 4467 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242594 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242598 4467 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242602 4467 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:35:21.260061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242607 4467 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.242653 4467 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242908 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242917 4467 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242920 4467 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242924 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242930 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242934 4467 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242939 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242943 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242954 4467 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242959 4467 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242965 4467 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242969 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242973 4467 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242977 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:35:21.260766 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242982 4467 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242986 4467 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242990 4467 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242994 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.242998 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243003 4467 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243007 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243021 4467 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243025 4467 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243028 4467 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243032 4467 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243037 4467 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243042 4467 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243047 4467 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243051 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243056 4467 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243060 4467 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243063 4467 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243067 4467 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243073 4467 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:35:21.261299 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243077 4467 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243081 4467 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243086 4467 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243089 4467 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243092 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243097 4467 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243101 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243106 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243114 4467 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243120 4467 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243125 4467 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243129 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243133 4467 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243135 4467 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243138 4467 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243141 4467 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243143 4467 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243146 4467 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243148 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:35:21.262124 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243151 4467 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243161 4467 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243166 4467 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243170 4467 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243175 4467 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243178 4467 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243183 4467 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243186 4467 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243190 4467 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243194 4467 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243198 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243203 4467 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243207 4467 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243211 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243215 4467 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243219 4467 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243222 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243224 4467 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243227 4467 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:35:21.262864 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243230 4467 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243234 4467 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243238 4467 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243251 4467 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243256 4467 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243261 4467 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243266 4467 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243270 4467 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243274 4467 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243278 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243283 4467 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243287 4467 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243291 4467 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243294 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243297 4467 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243306 4467 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243309 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243312 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243314 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243317 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:35:21.263546 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243320 4467 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243322 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243325 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243327 4467 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243330 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243333 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243336 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243338 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243342 4467 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243346 4467 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243351 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243356 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243360 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243364 4467 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.243409 4467 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243470 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 09:35:21.264212 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243478 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243484 4467 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243489 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243492 4467 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243495 4467 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243498 4467 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243500 4467 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243503 4467 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243506 4467 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243509 4467 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243512 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243514 4467 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243524 4467 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243529 4467 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243534 4467 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243539 4467 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243544 4467 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243549 4467 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243553 4467 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243556 4467 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 09:35:21.264852 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243558 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243561 4467 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243564 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243567 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243572 4467 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243580 4467 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243585 4467 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243590 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243593 4467 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243596 4467 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243600 4467 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243604 4467 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243608 4467 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243612 4467 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243617 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243621 4467 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243626 4467 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243631 4467 feature_gate.go:386] unrecognized feature gate: Example Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243635 4467 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243638 4467 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 09:35:21.265443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243640 4467 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243644 4467 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243648 4467 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243652 4467 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243655 4467 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243665 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243669 4467 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243674 4467 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243679 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243683 4467 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243687 4467 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243692 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243696 4467 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243701 4467 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243707 4467 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243712 4467 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243716 4467 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243721 4467 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 09:35:21.266115 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243724 4467 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243729 4467 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243734 4467 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243739 4467 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243743 4467 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243748 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243752 4467 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243756 4467 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243760 4467 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243765 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243770 4467 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243775 4467 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243780 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243784 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243788 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243793 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243799 4467 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243803 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243808 4467 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243812 4467 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 09:35:21.266745 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243826 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243831 4467 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243836 4467 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243840 4467 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243844 4467 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243848 4467 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243853 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243858 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243862 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243866 4467 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243870 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243875 4467 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243879 4467 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243884 4467 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243900 4467 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243904 4467 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243909 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243915 4467 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243919 4467 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243923 4467 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 09:35:21.267615 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243927 4467 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243929 4467 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243932 4467 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243935 4467 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243938 4467 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243942 4467 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:21.243948 4467 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.244039 4467 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.245399 4467 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.245465 4467 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.245741 4467 server.go:1014] "Starting client certificate rotation" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.246419 4467 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.247939 4467 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.248765 4467 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 09:29:41 +0000 UTC" deadline="2028-07-07 00:49:02.092861522 +0000 UTC" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.248822 4467 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14703h13m40.84405342s" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.248938 4467 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.251413 4467 log.go:25] "Validated CRI v1 runtime API" Nov 02 09:35:21.268370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.253092 4467 log.go:25] "Validated CRI v1 image API" Nov 02 09:35:21.269163 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.254019 4467 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 09:35:21.269163 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.255875 4467 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 97f9c2fe-476b-4347-af4a-2180e928b025:/dev/nvme0n1p3 e5d73044-bfe5-40d9-a32d-92b43fe42ae7:/dev/nvme0n1p4] Nov 02 09:35:21.269286 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.255918 4467 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/088a0edf852154cb50fd6b4212ed5500960c521610f290192bcc8cc46c133f28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/088a0edf852154cb50fd6b4212ed5500960c521610f290192bcc8cc46c133f28/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a528e75759da5962195b15f2a8af2c13b6a548ac8c474800b5397339e9f657c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a528e75759da5962195b15f2a8af2c13b6a548ac8c474800b5397339e9f657c/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/183c6057c5038a7ae1a13f1dec45085349cbeb33a7894d27237f18b974967e94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/183c6057c5038a7ae1a13f1dec45085349cbeb33a7894d27237f18b974967e94/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ca635d5e0a4a93b3d5a9852a9cd69a495e962cbec4b0dfda7bc73a03e9a3147/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ca635d5e0a4a93b3d5a9852a9cd69a495e962cbec4b0dfda7bc73a03e9a3147/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21c65e7a8199528b63b2e795f25216edebf3e1a1097c8d3b97d8dea92c82616d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21c65e7a8199528b63b2e795f25216edebf3e1a1097c8d3b97d8dea92c82616d/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27b25240bd9a57bd88c4a821cb6619f53c1e58c50c6ce6714ebe46fbefe4e860/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27b25240bd9a57bd88c4a821cb6619f53c1e58c50c6ce6714ebe46fbefe4e860/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36c64add1ba65ba53cdfe82684b9e7a5fb2fc714414a7259b107861de0b19faa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36c64add1ba65ba53cdfe82684b9e7a5fb2fc714414a7259b107861de0b19faa/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41abe3a5e051d7f7cc31768290069bffbd09626962cb71427bd23b5a3937cf8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41abe3a5e051d7f7cc31768290069bffbd09626962cb71427bd23b5a3937cf8d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cdde665f3f19c8c14fb425a853a6da18f2cec5d1e6b714a5baf1ea478240a26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cdde665f3f19c8c14fb425a853a6da18f2cec5d1e6b714a5baf1ea478240a26/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/620c54e56ffcf8e00930ba7dadbbfb83d4b409a164e8b019234cc0de3dee7fba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/620c54e56ffcf8e00930ba7dadbbfb83d4b409a164e8b019234cc0de3dee7fba/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72181ef10532878e4a5b4bea37d80745fab7748fa38dbd564b0e4794c99c0424/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72181ef10532878e4a5b4bea37d80745fab7748fa38dbd564b0e4794c99c0424/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab514a92ea3dab13ecfda9a6ee08276c96fc67135206921f4ee52ad6dfd2988a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab514a92ea3dab13ecfda9a6ee08276c96fc67135206921f4ee52ad6dfd2988a/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0b2dec5ab7f3602e42fd8dc5cd559dce30835814caced894672f06ddf91d5b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0b2dec5ab7f3602e42fd8dc5cd559dce30835814caced894672f06ddf91d5b9/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c954f2e92759af3ce9b778caf0d383dfde8d2fdd22329f064e5f4c6abd93482e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c954f2e92759af3ce9b778caf0d383dfde8d2fdd22329f064e5f4c6abd93482e/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3aa6bfc2b798908fbfab594de184bf21b3cc4598706a28bfea98faebe663534/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3aa6bfc2b798908fbfab594de184bf21b3cc4598706a28bfea98faebe663534/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06c7eba7-d115-4796-89a5-47a4b7af0789/volumes/kubernetes.io~projected/kube-api-access-d798b:{mountpoint:/var/lib/kubelet/pods/06c7eba7-d115-4796-89a5-47a4b7af0789/volumes/kubernetes.io~projected/kube-api-access-d798b major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e/volumes/kubernetes.io~projected/kube-api-access-vt6mc:{mountpoint:/var/lib/kubelet/pods/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e/volumes/kubernetes.io~projected/kube-api-access-vt6mc major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a880a57-b5db-4f8f-944f-741952727830/volumes/kubernetes.io~projected/kube-api-access-9ktbp:{mountpoint:/var/lib/kubelet/pods/2a880a57-b5db-4f8f-944f-741952727830/volumes/kubernetes.io~projected/kube-api-access-9ktbp major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f0d6935-5f2d-488d-9441-3dd6c418b832/volumes/kubernetes.io~projected/kube-api-access-cv85g:{mountpoint:/var/lib/kubelet/pods/2f0d6935-5f2d-488d-9441-3dd6c418b832/volumes/kubernetes.io~projected/kube-api-access-cv85g major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volumes/kubernetes.io~projected/kube-api-access-gjgk8:{mountpoint:/var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volumes/kubernetes.io~projected/kube-api-access-gjgk8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3267a968-7d5c-4392-a775-65f20929c5bc/volumes/kubernetes.io~projected/kube-api-access-fn55t:{mountpoint:/var/lib/kubelet/pods/3267a968-7d5c-4392-a775-65f20929c5bc/volumes/kubernetes.io~projected/kube-api-access-fn55t major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/565aea70-fb40-497b-9a8a-876abb6edb02/volumes/kubernetes.io~projected/kube-api-access-8vhk4:{mountpoint:/var/lib/kubelet/pods/565aea70-fb40-497b-9a8a-876abb6edb02/volumes/kubernetes.io~projected/kube-api-access-8vhk4 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/565aea70-fb40-497b-9a8a-876abb6edb02/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/565aea70-fb40-497b-9a8a-876abb6edb02/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/568d527d-7b27-4e41-a946-1893e04d2653/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/568d527d-7b27-4e41-a946-1893e04d2653/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a71ee3d-860c-4201-9be3-8fc7e7404cbe/volumes/kubernetes.io~projected/kube-api-access-c8w89:{mountpoint:/var/lib/kubelet/pods/5a71ee3d-860c-4201-9be3-8fc7e7404cbe/volumes/kubernetes.io~projected/kube-api-access-c8w89 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a71ee3d-860c-4201-9be3-8fc7e7404cbe/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/5a71ee3d-860c-4201-9be3-8fc7e7404cbe/volumes/kubernetes.io~secret/signing-key major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6be0dcba-1629-4bc9-9cc2-77e736cdbac3/volumes/kubernetes.io~projected/kube-api-access-2f7qh:{mountpoint:/var/lib/kubelet/pods/6be0dcba-1629-4bc9-9cc2-77e736cdbac3/volumes/kubernetes.io~projected/kube-api-access-2f7qh major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d1d3a9e-5aab-4313-9626-e0f7772fe862/volumes/kubernetes.io~projected/kube-api-access-xrkzz:{mountpoint:/var/lib/kubelet/pods/6d1d3a9e-5aab-4313-9626-e0f7772fe862/volumes/kubernetes.io~projected/kube-api-access-xrkzz major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8790c677-2d64-4973-a7e5-9ba0c04cc22d/volumes/kubernetes.io~projected/kube-api-access-tgq25:{mountpoint:/var/lib/kubelet/pods/8790c677-2d64-4973-a7e5-9ba0c04cc22d/volumes/kubernetes.io~projected/kube-api-access-tgq25 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc/volumes/kubernetes.io~projected/kube-api-access-7ph8j:{mountpoint:/var/lib/kubelet/pods/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc/volumes/kubernetes.io~projected/kube-api-access-7ph8j major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9db316d5-bf73-458b-ba54-9e4126840510/volumes/kubernetes.io~projected/kube-api-access-6v5cz:{mountpoint:/var/lib/kubelet/pods/9db316d5-bf73-458b-ba54-9e4126840510/volumes/kubernetes.io~projected/kube-api-access-6v5cz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e/volumes/kubernetes.io~projected/kube-api-access-l82bk:{mountpoint:/var/lib/kubelet/pods/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e/volumes/kubernetes.io~projected/kube-api-access-l82bk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc719dc6-3e71-4f85-b7fc-51cdee604e74/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/dc719dc6-3e71-4f85-b7fc-51cdee604e74/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~projected/kube-api-access-9fvkc:{mountpoint:/var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~projected/kube-api-access-9fvkc major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6d4f0ac824b61c4418129fa1f9ab8984c7c83e3cf5cd0b6e53b87a13ad341ce3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0bde4573e2599620a91658d005bead9671980725c32cc9d3cad4cfc047d4122a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d1b32a67e42d2eb979918d7fc69f960bc6632d76af6183e60e72eb55e4e0fdd1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/cddde0800b87167de2789335b0b847e740840ec782bbe07ab3f8a2092776e2ab/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4fd325c6de04888d9bdcf41971b6816c10879785bfc880be9f75e5e08b75a9a5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4e4f41f28db7dbd7d9f5afb3120b0808ec7e87802fa40813ca08f5752ffd7b6d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c67e34be8afc0bd3ddea38b47433fa1ff39d03ed1750027307e1613315d225f5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5d5123adc69802705c1a311dd9897ce544e67812990ca5295d561053c92ac1c3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b4b93d2e0fd844170396f66c07dc8bef55090ae750873994a0e2c9a8c067c887/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f17daeb5d9204e9ac9f198943451ccb27766f6bd4d7bdf3fcdaaf17b7ed881bd/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/345c99da4aeaa5888bbd39c7c8e1299002a25dec5b6bd8758954afb2da8b3def/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/7abb14adf90cfc45f89666c304f45abec64899e075806c11b74cc40a5a71c468/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a6070a8ba661058258fc163a45c53c341b7f2a54ada0afd6cff86f4bf67dfd8f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0c3cf628d57acef1e05c5c9285b964642546cbad45152e35fa9f4c8e63c6c4e1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6a2572c8cf66ea2c13a6d22b8e5c6cf9f36b22d627a220f61c11379f16619d40/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8da44a2cf427b5dd28c6e5c2eeea9888f6422f659221bbf95c3fc817cfc14621/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/24f1bc34562ca4a0a83911898c7954ffbb1fdcde2dd778e8cee60fa1271f07d9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d8695a366f856cf46867eabba0e090f7b60b9ca0ea5d33662ae295219dae94ed/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/77603b4bfa7118b5d8feb8d5acc962a7730096c92e2d4e02fad2f0da07236756/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d1ac4375b3f9f2f8d3a6b5f7a23ee10b1804cbff628c9536495cb092bc8ea6b6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/cf72d5541d515c8c6de7d7fbf881d62d22ee96da881d4420336729fdd7c2e080/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/17b208d9c4905f2302bb4967be22aab362b7b4c3f9145b8fbca89b61e2881cfd/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/6af04ea0d4fd107540fc275fa4cf16bcd65cfa7a38a015cb4c0714780a0ed5fe/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/7a14d52bddb7c07573b84bc71d4fdcb786e1c66d1de69c45332f543774d27781/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/2a0c0a7d77357f3a25d910bd34747f9738013cd0fc30de103af013ca7328988b/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/ffa0eebab181cdd81a373711cbb633a157db41ce3003a5da1c8c9c89887695a0/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/c5d4a266505dd858f7474705aa241725bf863b6d74f02ae557c61928e8fe1294/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/b449061e76edde6edae63117d5325b2ba9981bbe7ac2740def27fd206cca7e19/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/2860af560c157cab5798f5df2da593fe8f1096955f6480cf50cc983b9066404f/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/f2509c443fbeba98e994fcc6862896b652b79bf5190e5769fa25bccdf718619f/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/daf3b74f0d9969ad4c6f98345546a64ba8195879f83d790ce720ed302aa0e771/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/100ba716f454b3ed0da9473a1b4bbfb13ad7cd309864e8077511a0b7471ef556/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/9dd4179602c4c1a8416dceb3bc713a3fae640d63acfe1fe764ed8e384d6e4549/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/aadf6245b651c4d992c24eeb140915d4f18d0f8f3e5c58e8a1cd8358fa825197/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/35319c7ec59248439aec53016e0f08f375a8d633b1668d09e9ae7074ac484fd3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/1dac7035fe6c7dfaaa47fd352dcccee7db570fc5bd2823f248aa8bae4e820853/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/704ea17f130ce11face31434e6a4fb0c55cf5c0be532c9f4da92487d6d5492d2/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0edc828cd1ca7858b4f4dda21a3c00152f82c52dafce980f4fa300ddc73fbd26/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/aae9c3864f8bb01e9967f9f22661d97d9eb5b7ac85221b12ce8928db4970a9a2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/940a5f9d0d7b06d8084457536219249cfeab9dcca297ddb9afd954a302a9a5de/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.264108 4467 manager.go:217] Machine: {Timestamp:2026-11-02 09:35:21.263362845 +0000 UTC m=+0.103304836 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100172 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e383783e2893636ee295ad01c4bec SystemUUID:ec2e3837-83e2-8936-36ee-295ad01c4bec BootID:53e63310-996b-479f-a3e8-2b690d0f9b97 Filesystems:[{Device:/run/containers/storage/overlay-containers/27b25240bd9a57bd88c4a821cb6619f53c1e58c50c6ce6714ebe46fbefe4e860/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc/volumes/kubernetes.io~projected/kube-api-access-7ph8j DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/565aea70-fb40-497b-9a8a-876abb6edb02/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8790c677-2d64-4973-a7e5-9ba0c04cc22d/volumes/kubernetes.io~projected/kube-api-access-tgq25 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/565aea70-fb40-497b-9a8a-876abb6edb02/volumes/kubernetes.io~projected/kube-api-access-8vhk4 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21c65e7a8199528b63b2e795f25216edebf3e1a1097c8d3b97d8dea92c82616d/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cdde665f3f19c8c14fb425a853a6da18f2cec5d1e6b714a5baf1ea478240a26/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06c7eba7-d115-4796-89a5-47a4b7af0789/volumes/kubernetes.io~projected/kube-api-access-d798b DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2a880a57-b5db-4f8f-944f-741952727830/volumes/kubernetes.io~projected/kube-api-access-9ktbp DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab514a92ea3dab13ecfda9a6ee08276c96fc67135206921f4ee52ad6dfd2988a/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9db316d5-bf73-458b-ba54-9e4126840510/volumes/kubernetes.io~projected/kube-api-access-6v5cz DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ca635d5e0a4a93b3d5a9852a9cd69a495e962cbec4b0dfda7bc73a03e9a3147/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/2f0d6935-5f2d-488d-9441-3dd6c418b832/volumes/kubernetes.io~projected/kube-api-access-cv85g DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volumes/kubernetes.io~projected/kube-api-access-gjgk8 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a528e75759da5962195b15f2a8af2c13b6a548ac8c474800b5397339e9f657c/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/183c6057c5038a7ae1a13f1dec45085349cbeb33a7894d27237f18b974967e94/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36c64add1ba65ba53cdfe82684b9e7a5fb2fc714414a7259b107861de0b19faa/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/088a0edf852154cb50fd6b4212ed5500960c521610f290192bcc8cc46c133f28/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/620c54e56ffcf8e00930ba7dadbbfb83d4b409a164e8b019234cc0de3dee7fba/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6be0dcba-1629-4bc9-9cc2-77e736cdbac3/volumes/kubernetes.io~projected/kube-api-access-2f7qh DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e/volumes/kubernetes.io~projected/kube-api-access-vt6mc DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41abe3a5e051d7f7cc31768290069bffbd09626962cb71427bd23b5a3937cf8d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3aa6bfc2b798908fbfab594de184bf21b3cc4598706a28bfea98faebe663534/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc719dc6-3e71-4f85-b7fc-51cdee604e74/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72181ef10532878e4a5b4bea37d80745fab7748fa38dbd564b0e4794c99c0424/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0b2dec5ab7f3602e42fd8dc5cd559dce30835814caced894672f06ddf91d5b9/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f265c2b-4beb-4f7b-bed5-d91ef6c00377/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c954f2e92759af3ce9b778caf0d383dfde8d2fdd22329f064e5f4c6abd93482e/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e/volumes/kubernetes.io~projected/kube-api-access-l82bk DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3267a968-7d5c-4392-a775-65f20929c5bc/volumes/kubernetes.io~projected/kube-api-access-fn55t DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a71ee3d-860c-4201-9be3-8fc7e7404cbe/volumes/kubernetes.io~projected/kube-api-access-c8w89 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1/volumes/kubernetes.io~projected/kube-api-access-9fvkc DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/568d527d-7b27-4e41-a946-1893e04d2653/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a71ee3d-860c-4201-9be3-8fc7e7404cbe/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6d1d3a9e-5aab-4313-9626-e0f7772fe862/volumes/kubernetes.io~projected/kube-api-access-xrkzz DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1ca635d5e0a4a93 MacAddress:2e:e2:95:dc:a3:d5 Speed:10000 Mtu:8901} {Name:21c65e7a8199528 MacAddress:2a:cc:7e:2e:82:0d Speed:10000 Mtu:8901} {Name:27b25240bd9a57b MacAddress:d6:69:14:3b:de:f5 Speed:10000 Mtu:8901} {Name:72181ef10532878 MacAddress:7a:19:31:f3:d2:26 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c0:27:c9:3b Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:f9:e2:c7:69:5c Speed:0 Mtu:8901} {Name:c954f2e92759af3 MacAddress:6e:dd:ed:0a:d8:82 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c0:27:c9:3b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ce:3a:16:6f:65:e6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:02:16:e5:85:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.264483 4467 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.264556 4467 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.264930 4467 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.264962 4467 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-234.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265160 4467 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265173 4467 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265188 4467 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265214 4467 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265310 4467 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265511 4467 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265683 4467 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265746 4467 kubelet.go:493] "Attempting to sync node with API server" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265763 4467 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.265779 4467 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.266716 4467 kubelet.go:405] "Adding apiserver pod source" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.266762 4467 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.267613 4467 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.267626 4467 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 09:35:21.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.268186 4467 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 09:35:21.280764 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.280741 4467 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 09:35:21.280764 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.280758 4467 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 09:35:21.280948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.280906 4467 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 09:35:21.281160 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281146 4467 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 09:35:21.281213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281190 4467 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 09:35:21.281359 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281342 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 09:35:21.281412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281369 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 09:35:21.281412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281380 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 09:35:21.281412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281390 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 09:35:21.281412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281399 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 09:35:21.281412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281406 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 09:35:21.281623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281415 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 09:35:21.281623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281424 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 09:35:21.281623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281436 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 09:35:21.281623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281444 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 09:35:21.281623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281458 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 09:35:21.281623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281473 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 09:35:21.281623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281505 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 09:35:21.281623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.281514 4467 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 09:35:21.285719 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.285688 4467 server.go:1263] "Started kubelet" Nov 02 09:35:21.285988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.285934 4467 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 09:35:21.286435 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.286156 4467 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 09:35:21.286435 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.286243 4467 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 09:35:21.286273 ip-10-0-138-234 systemd[1]: Started Kubernetes Kubelet. Nov 02 09:35:21.287218 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.287200 4467 server.go:317] "Adding debug handlers to kubelet server" Nov 02 09:35:21.287472 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.287453 4467 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:35:21.289658 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.289640 4467 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 09:35:21.289813 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.289794 4467 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 09:35:21.290361 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.290334 4467 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:35:21.291385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.290724 4467 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:43 +0000 UTC" deadline="2028-07-08 22:05:19.435697913 +0000 UTC" Nov 02 09:35:21.291385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.290768 4467 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14748h29m58.144933545s" Nov 02 09:35:21.291385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.290804 4467 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 09:35:21.291385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.290812 4467 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 09:35:21.291385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.290917 4467 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 09:35:21.291385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.291192 4467 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 09:35:21.291883 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.291861 4467 factory.go:55] Registering systemd factory Nov 02 09:35:21.291983 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.291885 4467 factory.go:223] Registration of the systemd container factory successfully Nov 02 09:35:21.292676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292615 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ff82d1d-e5cc-4703-847e-1c62ea27ef2e" volumeName="kubernetes.io/empty-dir/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-data-volume" seLinuxMountContext="" Nov 02 09:35:21.292676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292654 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3267a968-7d5c-4392-a775-65f20929c5bc" volumeName="kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t" seLinuxMountContext="" Nov 02 09:35:21.292676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292672 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8790c677-2d64-4973-a7e5-9ba0c04cc22d" volumeName="kubernetes.io/configmap/8790c677-2d64-4973-a7e5-9ba0c04cc22d-config-volume" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292693 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a880a57-b5db-4f8f-944f-741952727830" volumeName="kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-multus-daemon-config" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292706 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f0d6935-5f2d-488d-9441-3dd6c418b832" volumeName="kubernetes.io/projected/2f0d6935-5f2d-488d-9441-3dd6c418b832-kube-api-access-cv85g" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292719 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f265c2b-4beb-4f7b-bed5-d91ef6c00377" volumeName="kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-env-overrides" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292735 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad130bd-a2d7-40a9-8d1e-d6a728b7d49e" volumeName="kubernetes.io/projected/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-kube-api-access-l82bk" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292748 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a71ee3d-860c-4201-9be3-8fc7e7404cbe" volumeName="kubernetes.io/secret/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-key" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292761 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1" volumeName="kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-tmp" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292775 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1" volumeName="kubernetes.io/projected/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-kube-api-access-9fvkc" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292792 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06c7eba7-d115-4796-89a5-47a4b7af0789" volumeName="kubernetes.io/configmap/06c7eba7-d115-4796-89a5-47a4b7af0789-serviceca" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292805 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ff82d1d-e5cc-4703-847e-1c62ea27ef2e" volumeName="kubernetes.io/projected/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-api-access-vt6mc" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292816 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f0d6935-5f2d-488d-9441-3dd6c418b832" volumeName="kubernetes.io/configmap/2f0d6935-5f2d-488d-9441-3dd6c418b832-iptables-alerter-script" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292827 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f265c2b-4beb-4f7b-bed5-d91ef6c00377" volumeName="kubernetes.io/projected/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-kube-api-access-gjgk8" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292841 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568d527d-7b27-4e41-a946-1893e04d2653" volumeName="kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292853 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d1d3a9e-5aab-4313-9626-e0f7772fe862" volumeName="kubernetes.io/projected/6d1d3a9e-5aab-4313-9626-e0f7772fe862-kube-api-access-xrkzz" seLinuxMountContext="" Nov 02 09:35:21.292857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292865 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8790c677-2d64-4973-a7e5-9ba0c04cc22d" volumeName="kubernetes.io/empty-dir/8790c677-2d64-4973-a7e5-9ba0c04cc22d-tmp-dir" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292877 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dcc0e95-455b-4a11-bb79-ff4e9d3338dc" volumeName="kubernetes.io/projected/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc-kube-api-access-7ph8j" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292907 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ff82d1d-e5cc-4703-847e-1c62ea27ef2e" volumeName="kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292919 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8790c677-2d64-4973-a7e5-9ba0c04cc22d" volumeName="kubernetes.io/projected/8790c677-2d64-4973-a7e5-9ba0c04cc22d-kube-api-access-tgq25" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292931 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db316d5-bf73-458b-ba54-9e4126840510" volumeName="kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292943 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad130bd-a2d7-40a9-8d1e-d6a728b7d49e" volumeName="kubernetes.io/empty-dir/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-tmp-dir" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292955 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a880a57-b5db-4f8f-944f-741952727830" volumeName="kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-cni-binary-copy" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292972 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a71ee3d-860c-4201-9be3-8fc7e7404cbe" volumeName="kubernetes.io/configmap/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-cabundle" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.292990 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a71ee3d-860c-4201-9be3-8fc7e7404cbe" volumeName="kubernetes.io/projected/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-kube-api-access-c8w89" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293003 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db316d5-bf73-458b-ba54-9e4126840510" volumeName="kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293021 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db316d5-bf73-458b-ba54-9e4126840510" volumeName="kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-binary-copy" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293037 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1" volumeName="kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-tuned" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293060 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06c7eba7-d115-4796-89a5-47a4b7af0789" volumeName="kubernetes.io/projected/06c7eba7-d115-4796-89a5-47a4b7af0789-kube-api-access-d798b" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293074 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f265c2b-4beb-4f7b-bed5-d91ef6c00377" volumeName="kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-script-lib" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293088 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f265c2b-4beb-4f7b-bed5-d91ef6c00377" volumeName="kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-config" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293104 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565aea70-fb40-497b-9a8a-876abb6edb02" volumeName="kubernetes.io/projected/565aea70-fb40-497b-9a8a-876abb6edb02-kube-api-access-8vhk4" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293122 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565aea70-fb40-497b-9a8a-876abb6edb02" volumeName="kubernetes.io/secret/565aea70-fb40-497b-9a8a-876abb6edb02-cert" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293138 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6be0dcba-1629-4bc9-9cc2-77e736cdbac3" volumeName="kubernetes.io/projected/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kube-api-access-2f7qh" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293149 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a880a57-b5db-4f8f-944f-741952727830" volumeName="kubernetes.io/projected/2a880a57-b5db-4f8f-944f-741952727830-kube-api-access-9ktbp" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293164 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f265c2b-4beb-4f7b-bed5-d91ef6c00377" volumeName="kubernetes.io/secret/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293180 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db316d5-bf73-458b-ba54-9e4126840510" volumeName="kubernetes.io/projected/9db316d5-bf73-458b-ba54-9e4126840510-kube-api-access-6v5cz" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293197 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc719dc6-3e71-4f85-b7fc-51cdee604e74" volumeName="kubernetes.io/configmap/dc719dc6-3e71-4f85-b7fc-51cdee604e74-konnectivity-ca" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293211 4467 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc719dc6-3e71-4f85-b7fc-51cdee604e74" volumeName="kubernetes.io/secret/dc719dc6-3e71-4f85-b7fc-51cdee604e74-agent-certs" seLinuxMountContext="" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293222 4467 reconstruct.go:97] "Volume reconstruction finished" Nov 02 09:35:21.293500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293232 4467 reconciler.go:29] "Reconciler: start to sync state" Nov 02 09:35:21.294910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293590 4467 factory.go:153] Registering CRI-O factory Nov 02 09:35:21.294910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.293608 4467 factory.go:223] Registration of the crio container factory successfully Nov 02 09:35:21.294910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.294134 4467 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 09:35:21.294910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.294178 4467 factory.go:103] Registering Raw factory Nov 02 09:35:21.294910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.294195 4467 manager.go:1199] Started watching for new ooms in manager Nov 02 09:35:21.294910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.294496 4467 manager.go:319] Starting recovery of all containers Nov 02 09:35:21.294910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.294705 4467 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:35:21.296649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.296626 4467 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 09:35:21.322452 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.322404 4467 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 09:35:21.324048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.324023 4467 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 09:35:21.324048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.324044 4467 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 09:35:21.324190 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.324064 4467 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 09:35:21.324190 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:21.324109 4467 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 09:35:21.328224 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.328201 4467 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 09:35:21.357027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.356954 4467 generic.go:356] "Generic (PLEG): container finished" podID="7dde2da7a6b8a26ba7501a7689619da1" containerID="9a38d4e29fbb7e1dc959d67e8d5e20123a92788ace7f891d79013b02ebf628db" exitCode=0 Nov 02 09:35:21.368437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.368407 4467 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="ba40d53444995891b189ed5b8193fdd5e4a0efd8af32bd802a7aefc3847b7afb" exitCode=0 Nov 02 09:35:21.368437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.368434 4467 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="0984e82e421729407267e32819438a92bbff4836b8dfd149f85bdd63ffc37d39" exitCode=0 Nov 02 09:35:21.368437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.368442 4467 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="f801ccb40abda1d067d48772e8358bfe3f42f9d749dd0215cca312263f8d9575" exitCode=0 Nov 02 09:35:21.368638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.368450 4467 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="f2ae6a3f095d12965708c92af4eb0c264b048f3dd5353cd49291a8c15cc0cbe8" exitCode=0 Nov 02 09:35:21.368638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.368458 4467 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="87ae2379d36f616fa363d9b922f42bc9cda5d109c898cf56f80ec263772a939d" exitCode=0 Nov 02 09:35:21.368638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.368464 4467 generic.go:356] "Generic (PLEG): container finished" podID="9db316d5-bf73-458b-ba54-9e4126840510" containerID="873cc7b5b241e06c3c80318ff73214b05d3f88de024280035630ebcf2dc86404" exitCode=0 Nov 02 09:35:21.376580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.376561 4467 manager.go:324] Recovery completed Nov 02 09:35:21.390263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390248 4467 cpu_manager.go:226] "Starting" policy="none" Nov 02 09:35:21.390263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390260 4467 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 09:35:21.390402 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390277 4467 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 09:35:21.390443 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390400 4467 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 09:35:21.390847 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390411 4467 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 09:35:21.390884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390850 4467 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 09:35:21.390884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390860 4467 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 09:35:21.390884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390868 4467 policy_none.go:50] "Start" Nov 02 09:35:21.390884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390877 4467 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 09:35:21.391011 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.390901 4467 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 09:35:21.391064 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.391053 4467 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 09:35:21.391098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.391064 4467 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 09:35:21.391098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.391070 4467 policy_none.go:44] "Start" Nov 02 09:35:21.395017 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.394992 4467 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 09:35:21.397504 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.397492 4467 manager.go:341] "Starting Device Plugin manager" Nov 02 09:35:21.397558 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:21.397514 4467 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 09:35:21.397558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.397522 4467 server.go:87] "Starting device plugin registration server" Nov 02 09:35:21.398353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.397978 4467 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 09:35:21.398353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.398007 4467 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 09:35:21.398353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.398155 4467 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 09:35:21.398353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.398286 4467 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 09:35:21.398353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.398358 4467 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 09:35:21.398643 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.398366 4467 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 09:35:21.398643 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.398430 4467 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T09:35:21.398341509Z","UUID":"f60fa35c-db5b-437c-be2c-048ec694a394","Handler":null,"Name":"","Endpoint":""} Nov 02 09:35:21.400067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.400046 4467 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 09:35:21.400165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.400077 4467 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 09:35:21.402243 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:21.402219 4467 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 09:35:21.424917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.424862 4467 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal"] Nov 02 09:35:21.425354 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.425321 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7ba5c7edc9657ad596c668f03e09721a","Type":"ContainerStarted","Data":"380ae39c5072f1d6d67326182093cbf2f04ed9dca5b19dd04f0fae338f7d11ec"} Nov 02 09:35:21.425448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.425362 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7ba5c7edc9657ad596c668f03e09721a","Type":"ContainerStarted","Data":"4cdde665f3f19c8c14fb425a853a6da18f2cec5d1e6b714a5baf1ea478240a26"} Nov 02 09:35:21.425448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.425398 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7dde2da7a6b8a26ba7501a7689619da1","Type":"ContainerStarted","Data":"532a5a1ce23baba89e054e4dcda74908e291ee7c48c0762cb91ca809d3725218"} Nov 02 09:35:21.425448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.425413 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7dde2da7a6b8a26ba7501a7689619da1","Type":"ContainerDied","Data":"9a38d4e29fbb7e1dc959d67e8d5e20123a92788ace7f891d79013b02ebf628db"} Nov 02 09:35:21.425448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.425426 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" event={"ID":"7dde2da7a6b8a26ba7501a7689619da1","Type":"ContainerStarted","Data":"620c54e56ffcf8e00930ba7dadbbfb83d4b409a164e8b019234cc0de3dee7fba"} Nov 02 09:35:21.425448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.425434 4467 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.425662 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.425513 4467 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.437616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.437596 4467 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 09:35:21.437693 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:21.437633 4467 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.438173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.438159 4467 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 09:35:21.438240 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:21.438198 4467 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.497525 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.497493 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.497636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.497532 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.497636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.497557 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba5c7edc9657ad596c668f03e09721a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7ba5c7edc9657ad596c668f03e09721a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.498531 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.498514 4467 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 09:35:21.499755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.499738 4467 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 09:35:21.499825 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.499767 4467 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 09:35:21.499825 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.499784 4467 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-234.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 09:35:21.499825 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.499820 4467 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.516259 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.516239 4467 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.516319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.516287 4467 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.598748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.598710 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.598748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.598750 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.599008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.598771 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba5c7edc9657ad596c668f03e09721a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7ba5c7edc9657ad596c668f03e09721a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.599008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.598844 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.599008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.598911 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dde2da7a6b8a26ba7501a7689619da1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7dde2da7a6b8a26ba7501a7689619da1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:21.599008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:21.598953 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ba5c7edc9657ad596c668f03e09721a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal\" (UID: \"7ba5c7edc9657ad596c668f03e09721a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:22.267425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.267390 4467 apiserver.go:52] "Watching apiserver" Nov 02 09:35:22.278867 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.278837 4467 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 09:35:22.279881 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.279854 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-7tfw7","openshift-dns/dns-default-fffwx","openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m","openshift-network-operator/iptables-alerter-lhvv8","kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-pvfk5","kube-system/konnectivity-agent-8t5nd","openshift-multus/multus-additional-cni-plugins-v8f95","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq","openshift-service-ca/service-ca-5fffc97bdf-2wf4v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal","openshift-multus/multus-f52r7","kube-system/global-pull-secret-syncer-zxrg9","openshift-dns/node-resolver-4xcxp","openshift-image-registry/node-ca-6kvjb","openshift-ingress-canary/ingress-canary-8hd89","openshift-network-diagnostics/network-check-target-j4f9g","openshift-insights/insights-runtime-extractor-cs7kk","openshift-ovn-kubernetes/ovnkube-node-9n577"] Nov 02 09:35:22.281983 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.281962 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:22.284270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.284252 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.284610 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.284596 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.285058 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.285035 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.285963 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.285943 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.286406 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.286382 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.286529 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.286511 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.287767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.287754 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 09:35:22.287853 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.287840 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 09:35:22.287921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.287854 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.287981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.287918 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sbddj\"" Nov 02 09:35:22.287981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.287931 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9tqmn\"" Nov 02 09:35:22.287981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.287955 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 09:35:22.287981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.287959 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 09:35:22.287981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.287972 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9d4cl\"" Nov 02 09:35:22.288149 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.288034 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kdt4x\"" Nov 02 09:35:22.289731 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.289716 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbhf4\"" Nov 02 09:35:22.289883 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.289870 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zc5hs\"" Nov 02 09:35:22.289883 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.289884 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-982vl\"" Nov 02 09:35:22.290010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.289933 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-llzhk\"" Nov 02 09:35:22.290354 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.290342 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t9b8s\"" Nov 02 09:35:22.290875 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.290859 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 09:35:22.290956 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.290916 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7vqld\"" Nov 02 09:35:22.290956 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.290933 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrsh4\"" Nov 02 09:35:22.291122 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.291106 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gztjj\"" Nov 02 09:35:22.291169 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.291141 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 09:35:22.291327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.291315 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2j22\"" Nov 02 09:35:22.291424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.291412 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 09:35:22.291504 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.291489 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 09:35:22.291552 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.291543 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbp4s\"" Nov 02 09:35:22.291611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.291580 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 09:35:22.292484 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.292235 4467 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.292602 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.292624 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.292716 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.292882 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293063 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293108 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293162 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293260 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293325 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293456 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293562 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6hxzv\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293604 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293668 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293727 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293744 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293752 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293775 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293784 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293876 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293909 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.293985 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294002 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294116 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294138 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294154 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294164 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294187 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294208 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294228 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294132 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294269 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294328 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pzrq9\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294405 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 09:35:22.295083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.294707 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zwj6d\"" Nov 02 09:35:22.299375 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.299353 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 09:35:22.302616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302597 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-ovn\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.302696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302626 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-config\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.302696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302642 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv85g\" (UniqueName: \"kubernetes.io/projected/2f0d6935-5f2d-488d-9441-3dd6c418b832-kube-api-access-cv85g\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:35:22.302696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302658 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-conf-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.302696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302673 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-sys\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.302917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302726 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-sys-fs\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.302917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302753 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-env-overrides\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.302917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302810 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:22.302917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302835 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.302917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302859 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-script-lib\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.302917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302876 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-etc-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.302917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302908 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-bin\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.303221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.302939 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c7eba7-d115-4796-89a5-47a4b7af0789-host\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:35:22.303221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303078 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-config\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.303221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303094 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0d6935-5f2d-488d-9441-3dd6c418b832-host-slash\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:35:22.303221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303117 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-system-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.303221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303168 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-crio-socket\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.303221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303198 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-registration-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.303221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303200 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovnkube-script-lib\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.303221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303218 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-env-overrides\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303258 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303283 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303283 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303318 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303338 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-bin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303354 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-hosts-file\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303368 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-modprobe-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303384 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-tuned\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303408 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-cabundle\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303482 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-slash\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303497 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-tuned\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303543 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgk8\" (UniqueName: \"kubernetes.io/projected/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-kube-api-access-gjgk8\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303571 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-cabundle\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303574 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhk4\" (UniqueName: \"kubernetes.io/projected/565aea70-fb40-497b-9a8a-876abb6edb02-kube-api-access-8vhk4\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:22.303601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303608 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-kubelet-config\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303624 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303638 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-systemd\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303670 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-host\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303703 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq25\" (UniqueName: \"kubernetes.io/projected/8790c677-2d64-4973-a7e5-9ba0c04cc22d-kube-api-access-tgq25\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303722 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovn-node-metrics-cert\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303737 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303758 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5cz\" (UniqueName: \"kubernetes.io/projected/9db316d5-bf73-458b-ba54-9e4126840510-kube-api-access-6v5cz\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303786 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-socket-dir-parent\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303803 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/565aea70-fb40-497b-9a8a-876abb6edb02-cert\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303858 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303907 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8790c677-2d64-4973-a7e5-9ba0c04cc22d-config-volume\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303935 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-log-socket\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303961 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-kubelet\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303987 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303989 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-tmp-dir\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.303908 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-ovn-node-metrics-cert\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304059 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-lib-modules\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304058 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-tmp-dir\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304076 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/565aea70-fb40-497b-9a8a-876abb6edb02-cert\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304088 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-var-lib-kubelet\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304112 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-data-volume\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304154 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8790c677-2d64-4973-a7e5-9ba0c04cc22d-config-volume\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304165 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7qh\" (UniqueName: \"kubernetes.io/projected/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kube-api-access-2f7qh\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304183 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8w89\" (UniqueName: \"kubernetes.io/projected/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-kube-api-access-c8w89\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304209 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-etc-kubernetes\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304231 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-node-log\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304241 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-data-volume\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304256 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.304293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304298 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-system-cni-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304337 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304363 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-k8s-cni-cncf-io\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304393 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-multus\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304423 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-socket-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304456 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-var-lib-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304475 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f0d6935-5f2d-488d-9441-3dd6c418b832-iptables-alerter-script\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304500 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-netns\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304516 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-multus-daemon-config\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304529 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9db316d5-bf73-458b-ba54-9e4126840510-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304530 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304639 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304817 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-multus-daemon-config\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304829 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2f0d6935-5f2d-488d-9441-3dd6c418b832-iptables-alerter-script\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304843 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ph8j\" (UniqueName: \"kubernetes.io/projected/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc-kube-api-access-7ph8j\") pod \"migrator-66df5d496b-b792m\" (UID: \"8dcc0e95-455b-4a11-bb79-ff4e9d3338dc\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304861 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkzz\" (UniqueName: \"kubernetes.io/projected/6d1d3a9e-5aab-4313-9626-e0f7772fe862-kube-api-access-xrkzz\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304880 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktbp\" (UniqueName: \"kubernetes.io/projected/2a880a57-b5db-4f8f-944f-741952727830-kube-api-access-9ktbp\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304917 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.304959 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82bk\" (UniqueName: \"kubernetes.io/projected/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-kube-api-access-l82bk\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305053 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-tmp\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.305131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305141 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/568d527d-7b27-4e41-a946-1893e04d2653-original-pull-secret\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305192 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-tmp\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305227 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc719dc6-3e71-4f85-b7fc-51cdee604e74-konnectivity-ca\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305257 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c677-2d64-4973-a7e5-9ba0c04cc22d-tmp-dir\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305284 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-systemd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305307 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06c7eba7-d115-4796-89a5-47a4b7af0789-serviceca\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305331 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-cni-binary-copy\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305372 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-hostroot\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305401 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc719dc6-3e71-4f85-b7fc-51cdee604e74-konnectivity-ca\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305452 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-kubernetes\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305471 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-run\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305458 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8790c677-2d64-4973-a7e5-9ba0c04cc22d-tmp-dir\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305492 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305515 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a880a57-b5db-4f8f-944f-741952727830-cni-binary-copy\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305535 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-key\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305563 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-systemd-units\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305587 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-cnibin\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305610 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-cnibin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305644 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysconfig\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305692 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6mc\" (UniqueName: \"kubernetes.io/projected/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-api-access-vt6mc\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305696 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06c7eba7-d115-4796-89a5-47a4b7af0789-serviceca\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305754 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-signing-key\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305777 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc719dc6-3e71-4f85-b7fc-51cdee604e74-agent-certs\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305804 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d798b\" (UniqueName: \"kubernetes.io/projected/06c7eba7-d115-4796-89a5-47a4b7af0789-kube-api-access-d798b\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305869 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305915 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-netd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.305938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305938 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-conf\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305958 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.305987 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-device-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.306023 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc719dc6-3e71-4f85-b7fc-51cdee604e74-agent-certs\") pod \"konnectivity-agent-8t5nd\" (UID: \"dc719dc6-3e71-4f85-b7fc-51cdee604e74\") " pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.306026 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.306072 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-kubelet\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.306101 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.306128 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-multus-certs\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.306154 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-dbus\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.306181 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvkc\" (UniqueName: \"kubernetes.io/projected/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-kube-api-access-9fvkc\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.306822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.306209 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-netns\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.313572 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.313552 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv85g\" (UniqueName: \"kubernetes.io/projected/2f0d6935-5f2d-488d-9441-3dd6c418b832-kube-api-access-cv85g\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:35:22.313661 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.313642 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn55t\" (UniqueName: \"kubernetes.io/projected/3267a968-7d5c-4392-a775-65f20929c5bc-kube-api-access-fn55t\") pod \"network-check-target-j4f9g\" (UID: \"3267a968-7d5c-4392-a775-65f20929c5bc\") " pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:22.315381 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.315366 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhk4\" (UniqueName: \"kubernetes.io/projected/565aea70-fb40-497b-9a8a-876abb6edb02-kube-api-access-8vhk4\") pod \"ingress-canary-8hd89\" (UID: \"565aea70-fb40-497b-9a8a-876abb6edb02\") " pod="openshift-ingress-canary/ingress-canary-8hd89" Nov 02 09:35:22.315420 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.315394 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgk8\" (UniqueName: \"kubernetes.io/projected/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-kube-api-access-gjgk8\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.315603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.315584 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7qh\" (UniqueName: \"kubernetes.io/projected/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kube-api-access-2f7qh\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.319686 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.319668 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkzz\" (UniqueName: \"kubernetes.io/projected/6d1d3a9e-5aab-4313-9626-e0f7772fe862-kube-api-access-xrkzz\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:22.319735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.319660 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktbp\" (UniqueName: \"kubernetes.io/projected/2a880a57-b5db-4f8f-944f-741952727830-kube-api-access-9ktbp\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.320482 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.320467 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6mc\" (UniqueName: \"kubernetes.io/projected/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-kube-api-access-vt6mc\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.320545 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.320529 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5cz\" (UniqueName: \"kubernetes.io/projected/9db316d5-bf73-458b-ba54-9e4126840510-kube-api-access-6v5cz\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.322184 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.322165 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ph8j\" (UniqueName: \"kubernetes.io/projected/8dcc0e95-455b-4a11-bb79-ff4e9d3338dc-kube-api-access-7ph8j\") pod \"migrator-66df5d496b-b792m\" (UID: \"8dcc0e95-455b-4a11-bb79-ff4e9d3338dc\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b792m" Nov 02 09:35:22.323602 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.323587 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82bk\" (UniqueName: \"kubernetes.io/projected/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-kube-api-access-l82bk\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:35:22.323708 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.323693 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d798b\" (UniqueName: \"kubernetes.io/projected/06c7eba7-d115-4796-89a5-47a4b7af0789-kube-api-access-d798b\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:35:22.324048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.324031 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq25\" (UniqueName: \"kubernetes.io/projected/8790c677-2d64-4973-a7e5-9ba0c04cc22d-kube-api-access-tgq25\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.324855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.324840 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvkc\" (UniqueName: \"kubernetes.io/projected/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-kube-api-access-9fvkc\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.327589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.327570 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8w89\" (UniqueName: \"kubernetes.io/projected/5a71ee3d-860c-4201-9be3-8fc7e7404cbe-kube-api-access-c8w89\") pod \"service-ca-5fffc97bdf-2wf4v\" (UID: \"5a71ee3d-860c-4201-9be3-8fc7e7404cbe\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" Nov 02 09:35:22.380179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.380150 4467 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:22.380349 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.380281 4467 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:22.392517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.392495 4467 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 09:35:22.392573 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.392543 4467 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:22.395198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.395181 4467 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 09:35:22.395299 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.395224 4467 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-234.us-east-2.compute.internal" Nov 02 09:35:22.407187 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407159 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-etc-kubernetes\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407187 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407189 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-node-log\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407205 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407222 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-system-cni-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407238 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-k8s-cni-cncf-io\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407266 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-etc-kubernetes\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407279 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-node-log\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407271 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-multus\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407296 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-system-cni-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407306 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-multus\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407283 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407331 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-socket-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407374 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-k8s-cni-cncf-io\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407386 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-var-lib-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407400 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-socket-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407412 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-var-lib-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407417 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-netns\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407449 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-systemd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407475 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-systemd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407452 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-netns\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407475 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-hostroot\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407504 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-hostroot\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407524 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-kubernetes\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407544 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-run\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407567 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407592 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-systemd-units\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407616 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-cnibin\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407629 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-kubernetes\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407639 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-cnibin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407663 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-cnibin\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407645 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-systemd-units\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.407688 4467 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407731 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysconfig\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407639 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-run\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407685 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysconfig\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407693 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-cnibin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.407757 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.407767 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.907745619 +0000 UTC m=+1.747687615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : secret "insights-runtime-extractor-tls" not found Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407807 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407825 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-netd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407840 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-conf\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407859 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407876 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407879 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-device-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407918 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-netd\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407930 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407926 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-device-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407943 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407954 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-kubelet\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407970 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-conf\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407980 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.407984 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-etc-selinux\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408005 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-multus-certs\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408011 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-kubelet\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408022 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-dbus\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408047 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-netns\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408057 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-run-multus-certs\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408069 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9db316d5-bf73-458b-ba54-9e4126840510-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8f95\" (UID: \"9db316d5-bf73-458b-ba54-9e4126840510\") " pod="openshift-multus/multus-additional-cni-plugins-v8f95" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408073 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-ovn\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408091 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-ovn\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408113 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-conf-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408075 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-run-netns\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408138 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-sys\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408156 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-dbus\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408162 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-sys-fs\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408166 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-conf-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408191 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-etc-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408198 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-sys\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408217 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-bin\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408228 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-sys-fs\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408235 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c7eba7-d115-4796-89a5-47a4b7af0789-host\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408238 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-etc-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408250 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0d6935-5f2d-488d-9441-3dd6c418b832-host-slash\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408262 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-cni-bin\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408268 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-system-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408275 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06c7eba7-d115-4796-89a5-47a4b7af0789-host\") pod \"node-ca-6kvjb\" (UID: \"06c7eba7-d115-4796-89a5-47a4b7af0789\") " pod="openshift-image-registry/node-ca-6kvjb" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408281 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0d6935-5f2d-488d-9441-3dd6c418b832-host-slash\") pod \"iptables-alerter-lhvv8\" (UID: \"2f0d6935-5f2d-488d-9441-3dd6c418b832\") " pod="openshift-network-operator/iptables-alerter-lhvv8" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408287 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-crio-socket\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408306 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-system-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408313 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-registration-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408338 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408315 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-crio-socket\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408362 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408364 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6be0dcba-1629-4bc9-9cc2-77e736cdbac3-registration-dir\") pod \"aws-ebs-csi-driver-node-tr8mq\" (UID: \"6be0dcba-1629-4bc9-9cc2-77e736cdbac3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tr8mq" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408387 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408412 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-bin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408413 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-run-openvswitch\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.408424 4467 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408458 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-cni-bin\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408463 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-cni-dir\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.408505 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.908474493 +0000 UTC m=+1.748416482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408523 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-hosts-file\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408547 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-modprobe-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408573 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-slash\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408600 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-kubelet-config\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408610 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aad130bd-a2d7-40a9-8d1e-d6a728b7d49e-hosts-file\") pod \"node-resolver-4xcxp\" (UID: \"aad130bd-a2d7-40a9-8d1e-d6a728b7d49e\") " pod="openshift-dns/node-resolver-4xcxp" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408623 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408642 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-host-slash\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408649 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-systemd\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408666 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/568d527d-7b27-4e41-a946-1893e04d2653-kubelet-config\") pod \"global-pull-secret-syncer-zxrg9\" (UID: \"568d527d-7b27-4e41-a946-1893e04d2653\") " pod="kube-system/global-pull-secret-syncer-zxrg9" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408671 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-host\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408685 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-modprobe-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408709 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-systemd\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408718 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-host\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408717 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-socket-dir-parent\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408746 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-multus-socket-dir-parent\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408753 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-etc-sysctl-d\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408755 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408790 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-log-socket\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408815 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-kubelet\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.408822 4467 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408837 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-lib-modules\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408839 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f265c2b-4beb-4f7b-bed5-d91ef6c00377-log-socket\") pod \"ovnkube-node-9n577\" (UID: \"2f265c2b-4beb-4f7b-bed5-d91ef6c00377\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:22.408794 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.408856 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.908844404 +0000 UTC m=+1.748786396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : secret "metrics-daemon-secret" not found Nov 02 09:35:22.411012 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408872 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a880a57-b5db-4f8f-944f-741952727830-host-var-lib-kubelet\") pod \"multus-f52r7\" (UID: \"2a880a57-b5db-4f8f-944f-741952727830\") " pod="openshift-multus/multus-f52r7" Nov 02 09:35:22.411012 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408915 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-var-lib-kubelet\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.411012 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408938 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-lib-modules\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.411012 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.408955 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1-var-lib-kubelet\") pod \"tuned-7tfw7\" (UID: \"dc7faf99-47b5-4aa1-8af8-6d4ba29fa6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-7tfw7" Nov 02 09:35:22.437757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.437723 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:22.586999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.586980 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 09:35:22.621716 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.621684 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:22.803266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.803238 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:22.912933 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.912899 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:22.913108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.912941 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:22.913108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.912975 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:22.913108 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.913037 4467 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:22.913108 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.913065 4467 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 09:35:22.913108 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.913082 4467 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 09:35:22.913253 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.913112 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:23.913095326 +0000 UTC m=+2.753037300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : secret "insights-runtime-extractor-tls" not found Nov 02 09:35:22.913253 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.913131 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:23.91312548 +0000 UTC m=+2.753067452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:22.913253 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:22.913141 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:23.91313599 +0000 UTC m=+2.753077963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : secret "metrics-daemon-secret" not found Nov 02 09:35:22.946034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:22.946007 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4xcxp_aad130bd-a2d7-40a9-8d1e-d6a728b7d49e/dns-node-resolver/0.log" Nov 02 09:35:23.384233 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:23.384200 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2wf4v" event={"ID":"5a71ee3d-860c-4201-9be3-8fc7e7404cbe","Type":"ContainerStarted","Data":"a00f736c4d651b54c3333e420ca95a0682d0057fbefd0b46b523eb95638b75a9"} Nov 02 09:35:23.918589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:23.918521 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:23.918589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:23.918592 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:23.918956 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:23.918630 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:23.918956 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:23.918761 4467 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 09:35:23.918956 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:23.918829 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:25.918810763 +0000 UTC m=+4.758752753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : secret "metrics-daemon-secret" not found Nov 02 09:35:23.919255 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:23.919238 4467 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 09:35:23.919321 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:23.919290 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:25.919273826 +0000 UTC m=+4.759215818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : secret "insights-runtime-extractor-tls" not found Nov 02 09:35:23.919374 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:23.919339 4467 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:23.919374 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:23.919367 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:25.91935789 +0000 UTC m=+4.759299883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:23.946311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:23.946285 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6kvjb_06c7eba7-d115-4796-89a5-47a4b7af0789/node-ca/0.log" Nov 02 09:35:24.077648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:24.077614 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:24.254437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:24.254348 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:24.268632 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:24.268598 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:24.436688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:24.436653 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:24.746214 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:24.746181 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8hd89_565aea70-fb40-497b-9a8a-876abb6edb02/serve-healthcheck-canary/0.log" Nov 02 09:35:24.821768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:24.821730 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:25.230462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:25.230412 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:25.346839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:25.346811 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-b792m_8dcc0e95-455b-4a11-bb79-ff4e9d3338dc/migrator/0.log" Nov 02 09:35:25.673547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:25.673500 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:25.931721 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:25.931621 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:25.931721 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:25.931687 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:25.931721 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:25.931717 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:25.931977 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:25.931785 4467 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 09:35:25.931977 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:25.931827 4467 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 09:35:25.931977 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:25.931869 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs podName:6d1d3a9e-5aab-4313-9626-e0f7772fe862 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:29.931852745 +0000 UTC m=+8.771794718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs") pod "network-metrics-daemon-pvfk5" (UID: "6d1d3a9e-5aab-4313-9626-e0f7772fe862") : secret "metrics-daemon-secret" not found Nov 02 09:35:25.931977 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:25.931800 4467 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:25.931977 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:25.931915 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls podName:0ff82d1d-e5cc-4703-847e-1c62ea27ef2e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:29.931880402 +0000 UTC m=+8.771822397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cs7kk" (UID: "0ff82d1d-e5cc-4703-847e-1c62ea27ef2e") : secret "insights-runtime-extractor-tls" not found Nov 02 09:35:25.931977 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:25.931945 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:29.931934598 +0000 UTC m=+8.771876584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:26.024187 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:26.024135 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:26.216139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:26.216040 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:26.470505 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:26.470393 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:26.808280 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:26.808250 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:26.808628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:26.808427 4467 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 09:35:26.818699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:26.818674 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:27.618113 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:27.618074 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:28.015639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:28.015552 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:28.229985 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:28.229947 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:28.424661 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:28.424630 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:28.830948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:28.830911 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:29.632186 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:29.632153 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:29.958864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:29.958772 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:29.958864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:29.958831 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:29.958864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:29.958864 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:29.959112 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:29.958980 4467 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 09:35:29.959112 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:29.959033 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls podName:8790c677-2d64-4973-a7e5-9ba0c04cc22d nodeName:}" failed. No retries permitted until 2026-11-02 09:35:37.959021244 +0000 UTC m=+16.798963230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls") pod "dns-default-fffwx" (UID: "8790c677-2d64-4973-a7e5-9ba0c04cc22d") : secret "dns-default-metrics-tls" not found Nov 02 09:35:29.959230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:29.959210 4467 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 09:35:29.961292 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:29.961266 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ff82d1d-e5cc-4703-847e-1c62ea27ef2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cs7kk\" (UID: \"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e\") " pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:29.961383 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:29.961299 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1d3a9e-5aab-4313-9626-e0f7772fe862-metrics-certs\") pod \"network-metrics-daemon-pvfk5\" (UID: \"6d1d3a9e-5aab-4313-9626-e0f7772fe862\") " pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:30.026289 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.026256 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:30.082523 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.082493 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pvfk5" Nov 02 09:35:30.086904 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.086865 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cs7kk" Nov 02 09:35:30.231508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.231436 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:30.355013 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.354983 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pvfk5"] Nov 02 09:35:30.358246 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:30.358213 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1d3a9e_5aab_4313_9626_e0f7772fe862.slice/crio-c0005180bb26edec342059f0a73c6824f8c9ae4327f7ed7825eb1ba39e7a3b7e WatchSource:0}: Error finding container c0005180bb26edec342059f0a73c6824f8c9ae4327f7ed7825eb1ba39e7a3b7e: Status 404 returned error can't find the container with id c0005180bb26edec342059f0a73c6824f8c9ae4327f7ed7825eb1ba39e7a3b7e Nov 02 09:35:30.358379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.358258 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cs7kk"] Nov 02 09:35:30.363699 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:30.363675 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff82d1d_e5cc_4703_847e_1c62ea27ef2e.slice/crio-48fd4655554b9c296fd8855a03195ee0fc9ab2921982ec84518dbda28d9c8538 WatchSource:0}: Error finding container 48fd4655554b9c296fd8855a03195ee0fc9ab2921982ec84518dbda28d9c8538: Status 404 returned error can't find the container with id 48fd4655554b9c296fd8855a03195ee0fc9ab2921982ec84518dbda28d9c8538 Nov 02 09:35:30.402803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.402774 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cs7kk" event={"ID":"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e","Type":"ContainerStarted","Data":"48fd4655554b9c296fd8855a03195ee0fc9ab2921982ec84518dbda28d9c8538"} Nov 02 09:35:30.404059 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.404033 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pvfk5" event={"ID":"6d1d3a9e-5aab-4313-9626-e0f7772fe862","Type":"ContainerStarted","Data":"c0005180bb26edec342059f0a73c6824f8c9ae4327f7ed7825eb1ba39e7a3b7e"} Nov 02 09:35:30.434345 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.434309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:30.708179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.708149 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:30.892149 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:30.892060 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:31.074547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:31.074512 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:31.258795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:31.258763 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:31.412185 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:31.412148 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cs7kk" event={"ID":"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e","Type":"ContainerStarted","Data":"509b0a99eb735a3e26769e5d37095979770e6fe270775ddfff199b1d688d5cf0"} Nov 02 09:35:31.412424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:31.412225 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cs7kk" event={"ID":"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e","Type":"ContainerStarted","Data":"87edf3905c533d0f1b1fbb2441fa1434d6fd10c25e2937902d59d2e74465a473"} Nov 02 09:35:31.414422 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:31.414382 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pvfk5" event={"ID":"6d1d3a9e-5aab-4313-9626-e0f7772fe862","Type":"ContainerStarted","Data":"909aa9b789d6c6fff442ebd3976b167d7a15fd5f929b5bfa048cb70199a77f38"} Nov 02 09:35:31.447378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:31.447346 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:32.018835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:32.018799 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:32.198920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:32.198856 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:32.384005 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:32.383965 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:32.419423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:32.419393 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pvfk5" event={"ID":"6d1d3a9e-5aab-4313-9626-e0f7772fe862","Type":"ContainerStarted","Data":"11961876bc2a1122a087cdb27acc5eacdb641850f34a6b882f4b9c88575d9e83"} Nov 02 09:35:32.543571 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:32.543539 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:32.626497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:32.626462 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:32.825026 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:32.824991 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:33.425205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:33.425171 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cs7kk" event={"ID":"0ff82d1d-e5cc-4703-847e-1c62ea27ef2e","Type":"ContainerStarted","Data":"ad83e0ce64d56d5bfe6421c21c76f8386e62c6cbafba45f4e33c00c1906b02b9"} Nov 02 09:35:33.427288 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:33.427267 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:33.779658 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:33.779572 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:34.020692 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:34.020655 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:34.229050 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:34.229017 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:34.623848 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:34.623819 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:34.827626 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:34.827596 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:35.017064 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:35.016958 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:35.197688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:35.197658 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:35.376071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:35.376041 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:37.623685 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.623656 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:37.719586 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.719543 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b"] Nov 02 09:35:37.722878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.722853 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" Nov 02 09:35:37.741934 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.738579 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 09:35:37.741934 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.738828 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hzzvs\"" Nov 02 09:35:37.741934 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.740037 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b"] Nov 02 09:35:37.813962 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.813924 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/903de960-6f67-4baa-95ac-6a6712998fc4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q2q2b\" (UID: \"903de960-6f67-4baa-95ac-6a6712998fc4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" Nov 02 09:35:37.819877 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.819840 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:37.915339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.915229 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/903de960-6f67-4baa-95ac-6a6712998fc4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q2q2b\" (UID: \"903de960-6f67-4baa-95ac-6a6712998fc4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" Nov 02 09:35:37.917779 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:37.917754 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/903de960-6f67-4baa-95ac-6a6712998fc4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q2q2b\" (UID: \"903de960-6f67-4baa-95ac-6a6712998fc4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" Nov 02 09:35:38.014975 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:38.014924 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:38.015872 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:38.015850 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:38.018169 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:38.018139 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8790c677-2d64-4973-a7e5-9ba0c04cc22d-metrics-tls\") pod \"dns-default-fffwx\" (UID: \"8790c677-2d64-4973-a7e5-9ba0c04cc22d\") " pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:38.034087 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:38.034054 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" Nov 02 09:35:38.185682 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:38.185596 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:38.307489 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:38.307407 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b"] Nov 02 09:35:38.311448 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:38.311422 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903de960_6f67_4baa_95ac_6a6712998fc4.slice/crio-3dc26f2502d811d8ea71aa8faf29b9f6afccebfecd7c6766dcadb8014d5225dd WatchSource:0}: Error finding container 3dc26f2502d811d8ea71aa8faf29b9f6afccebfecd7c6766dcadb8014d5225dd: Status 404 returned error can't find the container with id 3dc26f2502d811d8ea71aa8faf29b9f6afccebfecd7c6766dcadb8014d5225dd Nov 02 09:35:38.439744 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:38.439666 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" event={"ID":"903de960-6f67-4baa-95ac-6a6712998fc4","Type":"ContainerStarted","Data":"3dc26f2502d811d8ea71aa8faf29b9f6afccebfecd7c6766dcadb8014d5225dd"} Nov 02 09:35:38.457192 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:38.457165 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fffwx"] Nov 02 09:35:38.460244 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:38.460218 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8790c677_2d64_4973_a7e5_9ba0c04cc22d.slice/crio-41d070015152c6956d039245a760f3f5024410f8cc0247fccb5137e550d42f74 WatchSource:0}: Error finding container 41d070015152c6956d039245a760f3f5024410f8cc0247fccb5137e550d42f74: Status 404 returned error can't find the container with id 41d070015152c6956d039245a760f3f5024410f8cc0247fccb5137e550d42f74 Nov 02 09:35:39.445034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.444763 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" event={"ID":"903de960-6f67-4baa-95ac-6a6712998fc4","Type":"ContainerStarted","Data":"1d3c5cca9267093c50efbe2a353c335e6a163811e9860590119f0389eab2067f"} Nov 02 09:35:39.445034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.444941 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" Nov 02 09:35:39.446641 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.446612 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fffwx" event={"ID":"8790c677-2d64-4973-a7e5-9ba0c04cc22d","Type":"ContainerStarted","Data":"41d070015152c6956d039245a760f3f5024410f8cc0247fccb5137e550d42f74"} Nov 02 09:35:39.450292 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.450273 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" Nov 02 09:35:39.482156 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.482093 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2q2b" podStartSLOduration=1.461464568 podStartE2EDuration="2.482077101s" totalImagesPullingTime="1.020612533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:39.463045495 +0000 UTC m=+18.302987488" watchObservedRunningTime="2026-11-02 09:35:39.482077101 +0000 UTC m=+18.322019097" Nov 02 09:35:39.876637 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.876589 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8"] Nov 02 09:35:39.880016 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.879990 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:39.886673 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.886608 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kqngj\"" Nov 02 09:35:39.889235 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.889162 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 09:35:39.889628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.889608 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 09:35:39.890337 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.890312 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8"] Nov 02 09:35:39.890681 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.890662 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 09:35:39.890800 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.890713 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 09:35:39.930233 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.930198 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:39.930233 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.930239 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ckf\" (UniqueName: \"kubernetes.io/projected/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-kube-api-access-b6ckf\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:39.930479 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.930284 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:39.930479 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.930322 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:39.972055 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.972020 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:39.987185 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.987150 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 09:35:39.989909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:39.989867 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8t5nd" Nov 02 09:35:40.030694 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.030656 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.030905 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.030699 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ckf\" (UniqueName: \"kubernetes.io/projected/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-kube-api-access-b6ckf\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.030905 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.030738 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.030905 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.030767 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.032030 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.031624 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.032030 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:40.031740 4467 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Nov 02 09:35:40.032030 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:40.031808 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-tls podName:c364e8ee-e971-43c9-ab10-1e92f7ccddf8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:40.531790016 +0000 UTC m=+19.371731995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-tls") pod "prometheus-operator-9cd89b74c-x2bp8" (UID: "c364e8ee-e971-43c9-ab10-1e92f7ccddf8") : secret "prometheus-operator-tls" not found Nov 02 09:35:40.033569 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.033548 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.044348 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.044320 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ckf\" (UniqueName: \"kubernetes.io/projected/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-kube-api-access-b6ckf\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.159408 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.159325 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:40.360473 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.360446 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:40.460726 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.460683 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fffwx" event={"ID":"8790c677-2d64-4973-a7e5-9ba0c04cc22d","Type":"ContainerStarted","Data":"c323ab0f4fb80bd3936d7f1f2112cfe10940ebb8daca2fb80244a1594596413d"} Nov 02 09:35:40.534393 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.534359 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.536969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.536909 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c364e8ee-e971-43c9-ab10-1e92f7ccddf8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x2bp8\" (UID: \"c364e8ee-e971-43c9-ab10-1e92f7ccddf8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:40.796747 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:40.796663 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" Nov 02 09:35:41.069294 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:41.069261 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8"] Nov 02 09:35:41.071945 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:41.071914 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc364e8ee_e971_43c9_ab10_1e92f7ccddf8.slice/crio-ea284bffc9c0eddb0f2603f49864fcb3ff9689469750aca96a1399a870775105 WatchSource:0}: Error finding container ea284bffc9c0eddb0f2603f49864fcb3ff9689469750aca96a1399a870775105: Status 404 returned error can't find the container with id ea284bffc9c0eddb0f2603f49864fcb3ff9689469750aca96a1399a870775105 Nov 02 09:35:41.465035 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:41.464954 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" event={"ID":"c364e8ee-e971-43c9-ab10-1e92f7ccddf8","Type":"ContainerStarted","Data":"ea284bffc9c0eddb0f2603f49864fcb3ff9689469750aca96a1399a870775105"} Nov 02 09:35:41.466926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:41.466880 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fffwx" event={"ID":"8790c677-2d64-4973-a7e5-9ba0c04cc22d","Type":"ContainerStarted","Data":"eb7087c9f2932d8365f12cb0cba233cd30ddf304be794b28790b08ad7fcd899b"} Nov 02 09:35:42.472033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:42.471998 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" event={"ID":"c364e8ee-e971-43c9-ab10-1e92f7ccddf8","Type":"ContainerStarted","Data":"d28f6f89981e6a3898aacc17da30332a44c3482bc7e9642624e27cd29304fddf"} Nov 02 09:35:42.472443 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:42.472057 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" event={"ID":"c364e8ee-e971-43c9-ab10-1e92f7ccddf8","Type":"ContainerStarted","Data":"b72302f49e7a2c9edde5f10f05b910a1d2d802801013f1e6da11387724fed263"} Nov 02 09:35:42.472443 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:42.472132 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:42.492200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:42.492020 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x2bp8" podStartSLOduration=2.258443938 podStartE2EDuration="3.492002004s" totalImagesPullingTime="1.233558066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:42.4917466 +0000 UTC m=+21.331688594" watchObservedRunningTime="2026-11-02 09:35:42.492002004 +0000 UTC m=+21.331944000" Nov 02 09:35:42.722837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:42.722800 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:44.126908 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.126859 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:44.266635 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.266599 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4"] Nov 02 09:35:44.270322 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.270301 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.278803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.278780 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7ct4q"] Nov 02 09:35:44.282002 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.281977 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4"] Nov 02 09:35:44.282143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.282018 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.337649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.337617 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:44.341687 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.341661 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 09:35:44.341857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.341714 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-n7czp\"" Nov 02 09:35:44.341857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.341755 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 09:35:44.341857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.341786 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-phbth\"" Nov 02 09:35:44.341857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.341802 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 09:35:44.341857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.341815 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 09:35:44.361278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361251 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-wtmp\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.361431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361279 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-accelerators-collector-config\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.361431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361305 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f6ec3b03-947f-4658-a099-f3957b455739-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.361431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361390 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-root\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.361616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361431 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25b935c-ea6d-472f-a465-bb9b626c5365-metrics-client-ca\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.361616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361502 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4nc\" (UniqueName: \"kubernetes.io/projected/b25b935c-ea6d-472f-a465-bb9b626c5365-kube-api-access-bk4nc\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.361616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361595 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.361754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361643 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-sys\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.361754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361676 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.361845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361794 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.361845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361824 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-textfile\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.361948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361851 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6ec3b03-947f-4658-a099-f3957b455739-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.361948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361908 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 09:35:44.361948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361911 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr5qb\" (UniqueName: \"kubernetes.io/projected/f6ec3b03-947f-4658-a099-f3957b455739-kube-api-access-jr5qb\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.362073 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361953 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.362073 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.361972 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-tls\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.362073 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.362009 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 09:35:44.463010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.462918 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.463010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.462964 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-tls\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.462986 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-wtmp\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463108 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-wtmp\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463108 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-accelerators-collector-config\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463161 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f6ec3b03-947f-4658-a099-f3957b455739-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.463271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463191 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-root\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463220 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25b935c-ea6d-472f-a465-bb9b626c5365-metrics-client-ca\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463248 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4nc\" (UniqueName: \"kubernetes.io/projected/b25b935c-ea6d-472f-a465-bb9b626c5365-kube-api-access-bk4nc\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463277 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463295 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-root\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463323 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-sys\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463355 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463387 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463429 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-textfile\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463465 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6ec3b03-947f-4658-a099-f3957b455739-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463489 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr5qb\" (UniqueName: \"kubernetes.io/projected/f6ec3b03-947f-4658-a099-f3957b455739-kube-api-access-jr5qb\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.463564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463524 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f6ec3b03-947f-4658-a099-f3957b455739-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.464025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463598 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b25b935c-ea6d-472f-a465-bb9b626c5365-sys\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.464025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463789 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-accelerators-collector-config\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.464025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463807 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-textfile\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.464025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.463856 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.464402 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.464334 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6ec3b03-947f-4658-a099-f3957b455739-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.465143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.465117 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25b935c-ea6d-472f-a465-bb9b626c5365-metrics-client-ca\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.466043 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.466012 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-tls\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.466136 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.466070 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.466136 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.466116 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25b935c-ea6d-472f-a465-bb9b626c5365-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.466136 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.466127 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6ec3b03-947f-4658-a099-f3957b455739-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.475549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.475529 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4nc\" (UniqueName: \"kubernetes.io/projected/b25b935c-ea6d-472f-a465-bb9b626c5365-kube-api-access-bk4nc\") pod \"node-exporter-7ct4q\" (UID: \"b25b935c-ea6d-472f-a465-bb9b626c5365\") " pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.475859 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.475837 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr5qb\" (UniqueName: \"kubernetes.io/projected/f6ec3b03-947f-4658-a099-f3957b455739-kube-api-access-jr5qb\") pod \"kube-state-metrics-5dc8d7d89b-rg9l4\" (UID: \"f6ec3b03-947f-4658-a099-f3957b455739\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.517272 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.517239 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:44.589283 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.589250 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" Nov 02 09:35:44.599725 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.599691 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7ct4q" Nov 02 09:35:44.612397 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:44.612362 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25b935c_ea6d_472f_a465_bb9b626c5365.slice/crio-4147bfa66ead6f733d8d67e8e53ea1ab8f97ae7bd5299eade0f7e5aa5ca40770 WatchSource:0}: Error finding container 4147bfa66ead6f733d8d67e8e53ea1ab8f97ae7bd5299eade0f7e5aa5ca40770: Status 404 returned error can't find the container with id 4147bfa66ead6f733d8d67e8e53ea1ab8f97ae7bd5299eade0f7e5aa5ca40770 Nov 02 09:35:44.695554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.695520 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:44.869231 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.869198 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4"] Nov 02 09:35:44.872374 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:44.872350 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ec3b03_947f_4658_a099_f3957b455739.slice/crio-d3b838b0b5e2ab4b66162319997f83a92c9444adaf1d4968713dd8aecc8a6e6b WatchSource:0}: Error finding container d3b838b0b5e2ab4b66162319997f83a92c9444adaf1d4968713dd8aecc8a6e6b: Status 404 returned error can't find the container with id d3b838b0b5e2ab4b66162319997f83a92c9444adaf1d4968713dd8aecc8a6e6b Nov 02 09:35:44.876331 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:44.876301 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:45.341938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.341040 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 09:35:45.370956 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.370634 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.372468 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.372264 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 09:35:45.376314 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.376194 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 09:35:45.376314 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.376215 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 09:35:45.376932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.376909 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 09:35:45.377855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.377830 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 09:35:45.377978 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.377961 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-99zt2\"" Nov 02 09:35:45.378071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.378053 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 09:35:45.378191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.378139 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 09:35:45.378191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.378179 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 09:35:45.378298 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.378214 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 09:35:45.378350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.378323 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 09:35:45.471705 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.471661 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.471916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.471769 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.471916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.471812 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.471916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.471846 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-config-out\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.471916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.471881 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.472146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.471919 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.472146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.471944 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.472146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.471984 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.472146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.472008 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-config-volume\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.472146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.472026 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.472146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.472045 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mxp\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-kube-api-access-p7mxp\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.472146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.472073 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.472146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.472107 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-web-config\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.484134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.484094 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7ct4q" event={"ID":"b25b935c-ea6d-472f-a465-bb9b626c5365","Type":"ContainerStarted","Data":"4147bfa66ead6f733d8d67e8e53ea1ab8f97ae7bd5299eade0f7e5aa5ca40770"} Nov 02 09:35:45.485757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.485728 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" event={"ID":"f6ec3b03-947f-4658-a099-f3957b455739","Type":"ContainerStarted","Data":"d3b838b0b5e2ab4b66162319997f83a92c9444adaf1d4968713dd8aecc8a6e6b"} Nov 02 09:35:45.573445 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573410 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.573445 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573446 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.573699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573465 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.573699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573489 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.573699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573519 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-config-volume\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.573699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573548 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.573699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573572 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mxp\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-kube-api-access-p7mxp\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.573699 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:45.573615 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle podName:86fc3914-4c5e-4826-9484-0c84c891377e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:46.073592108 +0000 UTC m=+24.913534096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e") : configmap references non-existent config key: ca-bundle.crt Nov 02 09:35:45.573699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573660 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.574098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573711 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-web-config\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.574098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573749 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.574098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573781 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.574098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573810 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.574098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.573838 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-config-out\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.574098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.574035 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.575361 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.575305 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.577013 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.576967 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-web-config\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.577403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.577200 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-config-out\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.578447 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.578002 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.578447 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.578129 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.578447 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.578207 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.578447 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.578396 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.578447 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.578411 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-config-volume\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.578809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.578522 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.580091 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.580072 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:45.588377 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:45.588350 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mxp\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-kube-api-access-p7mxp\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:46.078239 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:46.078215 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:46.078830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:46.078810 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:46.335876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:46.335749 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:35:46.354854 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:46.354821 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:46.491243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:46.491208 4467 generic.go:356] "Generic (PLEG): container finished" podID="b25b935c-ea6d-472f-a465-bb9b626c5365" containerID="a80c03072ce386b2873c58501e7f164d7cbba440270a270903523aaa6962a798" exitCode=0 Nov 02 09:35:46.491421 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:46.491299 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7ct4q" event={"ID":"b25b935c-ea6d-472f-a465-bb9b626c5365","Type":"ContainerDied","Data":"a80c03072ce386b2873c58501e7f164d7cbba440270a270903523aaa6962a798"} Nov 02 09:35:46.541034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:46.541008 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:46.634367 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:46.634338 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 09:35:46.637195 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:46.637170 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fc3914_4c5e_4826_9484_0c84c891377e.slice/crio-05d1d73996a289b362cb1e81d785d06f3bb7dfd69c17e7e534c713ad01d30646 WatchSource:0}: Error finding container 05d1d73996a289b362cb1e81d785d06f3bb7dfd69c17e7e534c713ad01d30646: Status 404 returned error can't find the container with id 05d1d73996a289b362cb1e81d785d06f3bb7dfd69c17e7e534c713ad01d30646 Nov 02 09:35:47.346274 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.346242 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:47.497654 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.497605 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7ct4q" event={"ID":"b25b935c-ea6d-472f-a465-bb9b626c5365","Type":"ContainerStarted","Data":"174f5bf1574e0fa0593f90ddfad363e94f29f05a5ee75b7df9ae83787ae15874"} Nov 02 09:35:47.498158 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.497670 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7ct4q" event={"ID":"b25b935c-ea6d-472f-a465-bb9b626c5365","Type":"ContainerStarted","Data":"503781637ae835e8981fbae1f3415d1aaaa4057a583d601c3f88c17eb159ef0b"} Nov 02 09:35:47.500561 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.500536 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" event={"ID":"f6ec3b03-947f-4658-a099-f3957b455739","Type":"ContainerStarted","Data":"4e4c1623fbc63f498dcdcddc5e74f520f0494026875616fe7818688bae38f8b3"} Nov 02 09:35:47.500674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.500586 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" event={"ID":"f6ec3b03-947f-4658-a099-f3957b455739","Type":"ContainerStarted","Data":"aea6de9e5641237fd249c9e21644d69f85a61b6ee7e42a38a61d8f614d27bbbf"} Nov 02 09:35:47.500674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.500605 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" event={"ID":"f6ec3b03-947f-4658-a099-f3957b455739","Type":"ContainerStarted","Data":"4697934c3a92440212aa54e3aff70548a64da197651f799c0ab598228514f6b8"} Nov 02 09:35:47.501858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.501803 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerStarted","Data":"05d1d73996a289b362cb1e81d785d06f3bb7dfd69c17e7e534c713ad01d30646"} Nov 02 09:35:47.522974 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.522918 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7ct4q" podStartSLOduration=2.000957415 podStartE2EDuration="3.522878297s" totalImagesPullingTime="1.453256051s" totalInitContainerRuntime="68.664831ms" isStatefulPod=true podCreationTimestamp="2026-11-02 09:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:47.521449802 +0000 UTC m=+26.361391797" watchObservedRunningTime="2026-11-02 09:35:47.522878297 +0000 UTC m=+26.362820293" Nov 02 09:35:47.530420 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.530379 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:47.543066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:47.543008 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rg9l4" podStartSLOduration=1.823763069 podStartE2EDuration="3.542994733s" totalImagesPullingTime="1.719231664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:47.542117036 +0000 UTC m=+26.382059034" watchObservedRunningTime="2026-11-02 09:35:47.542994733 +0000 UTC m=+26.382936727" Nov 02 09:35:48.026037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.025953 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n577" Nov 02 09:35:48.507191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.507156 4467 generic.go:356] "Generic (PLEG): container finished" podID="86fc3914-4c5e-4826-9484-0c84c891377e" containerID="52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81" exitCode=0 Nov 02 09:35:48.507690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.507245 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerDied","Data":"52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81"} Nov 02 09:35:48.664426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.664392 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5994556fdb-hmv5q"] Nov 02 09:35:48.670079 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.670043 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.677540 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.677447 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 09:35:48.677701 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.677652 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 09:35:48.678765 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.678636 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5994556fdb-hmv5q"] Nov 02 09:35:48.705306 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.705278 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 09:35:48.705522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.705506 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hmrjl\"" Nov 02 09:35:48.705597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.705522 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 09:35:48.705648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.705598 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9do3d65bq3a4u\"" Nov 02 09:35:48.801110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.801011 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-secret-metrics-server-tls\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.801110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.801060 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0d1336-1b3f-45dd-b201-1bf53156fe92-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.801110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.801093 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-secret-metrics-server-client-certs\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.801367 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.801213 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca0d1336-1b3f-45dd-b201-1bf53156fe92-metrics-server-audit-profiles\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.801367 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.801244 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-client-ca-bundle\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.801367 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.801265 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmcb7\" (UniqueName: \"kubernetes.io/projected/ca0d1336-1b3f-45dd-b201-1bf53156fe92-kube-api-access-pmcb7\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.801367 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.801322 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca0d1336-1b3f-45dd-b201-1bf53156fe92-audit-log\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.902362 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.902328 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0d1336-1b3f-45dd-b201-1bf53156fe92-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.902558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.902377 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-secret-metrics-server-client-certs\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.902558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.902403 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca0d1336-1b3f-45dd-b201-1bf53156fe92-metrics-server-audit-profiles\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.902558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.902425 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-client-ca-bundle\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.902558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.902547 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcb7\" (UniqueName: \"kubernetes.io/projected/ca0d1336-1b3f-45dd-b201-1bf53156fe92-kube-api-access-pmcb7\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.902778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.902621 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca0d1336-1b3f-45dd-b201-1bf53156fe92-audit-log\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.902778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.902667 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-secret-metrics-server-tls\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.903163 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.903132 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca0d1336-1b3f-45dd-b201-1bf53156fe92-audit-log\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.903333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.903312 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0d1336-1b3f-45dd-b201-1bf53156fe92-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.903641 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.903617 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca0d1336-1b3f-45dd-b201-1bf53156fe92-metrics-server-audit-profiles\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.904978 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.904955 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-secret-metrics-server-client-certs\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.905062 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.905048 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-client-ca-bundle\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.905177 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.905158 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca0d1336-1b3f-45dd-b201-1bf53156fe92-secret-metrics-server-tls\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.915390 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.915358 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcb7\" (UniqueName: \"kubernetes.io/projected/ca0d1336-1b3f-45dd-b201-1bf53156fe92-kube-api-access-pmcb7\") pod \"metrics-server-5994556fdb-hmv5q\" (UID: \"ca0d1336-1b3f-45dd-b201-1bf53156fe92\") " pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:48.983855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:48.983813 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:35:49.045114 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.045083 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl"] Nov 02 09:35:49.055654 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.055597 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:49.058333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.057996 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl"] Nov 02 09:35:49.059834 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.059811 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 09:35:49.060575 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.060537 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rqbf5\"" Nov 02 09:35:49.104062 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.104027 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gghk7\" (UniqueName: \"kubernetes.io/projected/c741a9ba-17b1-48fc-9d63-44938bb816eb-kube-api-access-gghk7\") pod \"monitoring-plugin-77f4f76594-2hvrl\" (UID: \"c741a9ba-17b1-48fc-9d63-44938bb816eb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:49.104062 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.104074 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c741a9ba-17b1-48fc-9d63-44938bb816eb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2hvrl\" (UID: \"c741a9ba-17b1-48fc-9d63-44938bb816eb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:49.173075 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.173042 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:49.205298 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.205261 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gghk7\" (UniqueName: \"kubernetes.io/projected/c741a9ba-17b1-48fc-9d63-44938bb816eb-kube-api-access-gghk7\") pod \"monitoring-plugin-77f4f76594-2hvrl\" (UID: \"c741a9ba-17b1-48fc-9d63-44938bb816eb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:49.205298 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.205303 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c741a9ba-17b1-48fc-9d63-44938bb816eb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2hvrl\" (UID: \"c741a9ba-17b1-48fc-9d63-44938bb816eb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:49.207724 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.207704 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c741a9ba-17b1-48fc-9d63-44938bb816eb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2hvrl\" (UID: \"c741a9ba-17b1-48fc-9d63-44938bb816eb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:49.217969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.217943 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gghk7\" (UniqueName: \"kubernetes.io/projected/c741a9ba-17b1-48fc-9d63-44938bb816eb-kube-api-access-gghk7\") pod \"monitoring-plugin-77f4f76594-2hvrl\" (UID: \"c741a9ba-17b1-48fc-9d63-44938bb816eb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:49.268584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.268550 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5994556fdb-hmv5q"] Nov 02 09:35:49.271617 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:49.271588 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0d1336_1b3f_45dd_b201_1bf53156fe92.slice/crio-a4ca73b6c408d8dc78f8c27014bbddddf145f8650bd4322318ab2c301ef79ec6 WatchSource:0}: Error finding container a4ca73b6c408d8dc78f8c27014bbddddf145f8650bd4322318ab2c301ef79ec6: Status 404 returned error can't find the container with id a4ca73b6c408d8dc78f8c27014bbddddf145f8650bd4322318ab2c301ef79ec6 Nov 02 09:35:49.360912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.360842 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:49.375354 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.375324 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:49.512239 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.512197 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" event={"ID":"ca0d1336-1b3f-45dd-b201-1bf53156fe92","Type":"ContainerStarted","Data":"a4ca73b6c408d8dc78f8c27014bbddddf145f8650bd4322318ab2c301ef79ec6"} Nov 02 09:35:49.652011 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:49.651945 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl"] Nov 02 09:35:49.654421 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:49.654395 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc741a9ba_17b1_48fc_9d63_44938bb816eb.slice/crio-1bd579a098e1567d0428a3111d66214b1ed8db6a03ff040a58af568c866d4b9c WatchSource:0}: Error finding container 1bd579a098e1567d0428a3111d66214b1ed8db6a03ff040a58af568c866d4b9c: Status 404 returned error can't find the container with id 1bd579a098e1567d0428a3111d66214b1ed8db6a03ff040a58af568c866d4b9c Nov 02 09:35:50.519978 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.519939 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" event={"ID":"c741a9ba-17b1-48fc-9d63-44938bb816eb","Type":"ContainerStarted","Data":"1bd579a098e1567d0428a3111d66214b1ed8db6a03ff040a58af568c866d4b9c"} Nov 02 09:35:50.595413 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.594314 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:35:50.602703 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.602659 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.608123 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.607695 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 09:35:50.608123 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.607856 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 09:35:50.608123 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.608063 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.609406 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.610696 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.610741 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.611065 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bjq15mnn2s3mt\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.611222 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.612114 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.612157 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.612190 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 09:35:50.613304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.612238 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dvlfc\"" Nov 02 09:35:50.613841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.613753 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 09:35:50.613841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.613813 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 09:35:50.619017 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.616353 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 09:35:50.619017 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.618201 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:35:50.619017 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.618447 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 09:35:50.692411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.692373 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:50.715839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.715803 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-config\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.715860 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.715964 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716022 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716195 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716054 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716195 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716129 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716195 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716168 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716200 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716237 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716274 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716304 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ghc\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-kube-api-access-c6ghc\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716327 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716343 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716365 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716396 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716421 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716452 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716480 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716505 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.716549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.716519 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817256 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817320 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817353 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ghc\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-kube-api-access-c6ghc\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817392 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817421 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817450 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817481 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817510 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817537 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817566 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817604 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817633 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817668 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-config\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817715 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817737 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817778 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817806 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817836 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817850 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.817917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.817862 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.819926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.818651 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.819926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.818940 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.819926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.819880 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.821273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.821249 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.821946 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.821905 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.822577 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.822553 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.823531 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.823492 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.824164 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.824143 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.824448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.824425 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.828768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.828748 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.829466 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.829441 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.829550 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.829466 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.829662 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.829619 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.830213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.830174 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-config\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.830213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.830202 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.830531 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.830507 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.830852 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.830828 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.831234 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.831212 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.832008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.831989 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.835313 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.835267 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ghc\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-kube-api-access-c6ghc\") pod \"prometheus-k8s-0\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:50.980374 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:50.980070 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:35:51.306148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:51.306117 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:35:51.310359 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:35:51.310320 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5193e406_14fa_46c0_adf6_63c3061817e2.slice/crio-ab42df41690bd6afd3cefc12357b6ccab84d4bc18b7e5fad6dfe67d52baecb5d WatchSource:0}: Error finding container ab42df41690bd6afd3cefc12357b6ccab84d4bc18b7e5fad6dfe67d52baecb5d: Status 404 returned error can't find the container with id ab42df41690bd6afd3cefc12357b6ccab84d4bc18b7e5fad6dfe67d52baecb5d Nov 02 09:35:51.524676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:51.524639 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerStarted","Data":"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446"} Nov 02 09:35:51.525079 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:51.524704 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerStarted","Data":"ab42df41690bd6afd3cefc12357b6ccab84d4bc18b7e5fad6dfe67d52baecb5d"} Nov 02 09:35:52.477321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.477296 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fffwx" Nov 02 09:35:52.538944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.538655 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerStarted","Data":"5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74"} Nov 02 09:35:52.538944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.538737 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerStarted","Data":"54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36"} Nov 02 09:35:52.538944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.538782 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerStarted","Data":"973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5"} Nov 02 09:35:52.538944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.538820 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerStarted","Data":"39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea"} Nov 02 09:35:52.538944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.538859 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerStarted","Data":"40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7"} Nov 02 09:35:52.540530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.540501 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" event={"ID":"ca0d1336-1b3f-45dd-b201-1bf53156fe92","Type":"ContainerStarted","Data":"b1bce3426301e55178784de7911444af4fd2bd31f28326e1e161267ee00f6606"} Nov 02 09:35:52.542188 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.542167 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" event={"ID":"c741a9ba-17b1-48fc-9d63-44938bb816eb","Type":"ContainerStarted","Data":"258f0fc1ad8bff80cc80db95a57f5cafef030ab1afb7fa4c28348aab898fd935"} Nov 02 09:35:52.542341 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.542323 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:52.543977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.543940 4467 generic.go:356] "Generic (PLEG): container finished" podID="5193e406-14fa-46c0-adf6-63c3061817e2" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" exitCode=0 Nov 02 09:35:52.544067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.544003 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerDied","Data":"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446"} Nov 02 09:35:52.546004 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.545980 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j4f9g" Nov 02 09:35:52.547741 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.547725 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" Nov 02 09:35:52.560819 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.560766 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" podStartSLOduration=1.9904545439999999 podStartE2EDuration="4.560750498s" totalImagesPullingTime="2.570295954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:52.559733392 +0000 UTC m=+31.399675388" watchObservedRunningTime="2026-11-02 09:35:52.560750498 +0000 UTC m=+31.400692494" Nov 02 09:35:52.628243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.628191 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2hvrl" podStartSLOduration=1.058761759 podStartE2EDuration="3.628176585s" totalImagesPullingTime="2.569414826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:52.627106699 +0000 UTC m=+31.467048699" watchObservedRunningTime="2026-11-02 09:35:52.628176585 +0000 UTC m=+31.468118579" Nov 02 09:35:52.632571 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.632544 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:52.814844 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:52.814813 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:54.434829 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:54.434793 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:54.559639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:54.559588 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerStarted","Data":"8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b"} Nov 02 09:35:54.595397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:54.595336 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.8144681929999997 podStartE2EDuration="9.595321742s" totalImagesPullingTime="5.76761372s" totalInitContainerRuntime="13.239829ms" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:54.593436886 +0000 UTC m=+33.433378881" watchObservedRunningTime="2026-11-02 09:35:54.595321742 +0000 UTC m=+33.435263733" Nov 02 09:35:54.622439 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:54.622403 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:56.577187 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:56.577156 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerStarted","Data":"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d"} Nov 02 09:35:56.577728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:56.577233 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerStarted","Data":"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8"} Nov 02 09:35:56.854387 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:35:56.854354 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:35:56.973598 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:56.973561 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:57.154399 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:57.154328 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:57.351752 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:57.351716 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:57.532796 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:57.532703 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:57.749434 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:57.749401 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:58.159025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:58.158978 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:58.354954 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:58.354918 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:59.592907 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:59.592868 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerStarted","Data":"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e"} Nov 02 09:35:59.593305 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:59.592953 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerStarted","Data":"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3"} Nov 02 09:35:59.593305 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:59.592981 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerStarted","Data":"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26"} Nov 02 09:35:59.593305 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:59.593019 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerStarted","Data":"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e"} Nov 02 09:35:59.636205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:59.636141 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.729743509 podStartE2EDuration="9.636121594s" totalImagesPullingTime="5.878502511s" totalInitContainerRuntime="27.875574ms" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:59.6347095 +0000 UTC m=+38.474651496" watchObservedRunningTime="2026-11-02 09:35:59.636121594 +0000 UTC m=+38.476063590" Nov 02 09:35:59.794945 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:59.794912 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:35:59.994269 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:35:59.994192 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:00.561774 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:00.561737 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:00.754588 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:00.754555 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:00.969320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:00.969241 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:00.981113 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:00.981079 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:01.156003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:01.155969 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:03.581464 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:03.581386 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:03.764564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:03.764531 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:07.798767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:07.798734 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:08.033262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:08.033233 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:08.293345 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:08.293310 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:08.475702 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:08.475674 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:08.988942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:08.988907 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:36:08.989345 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:08.989002 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:36:10.007567 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:10.007533 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:10.221737 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:10.221705 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:13.511732 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:13.511692 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:13.695149 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:13.695115 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:19.110768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:19.110730 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:19.297770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:19.297737 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:22.612058 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:22.612022 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:22.803876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:22.803819 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:26.855098 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:26.855065 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:36:28.989470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:28.989437 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:36:28.993263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:28.993236 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5994556fdb-hmv5q" Nov 02 09:36:30.228797 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:30.228743 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:30.247707 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:30.247677 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:30.723045 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:30.723019 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:33.152737 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:33.152705 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:33.341995 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:33.341952 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:34.116463 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:34.116430 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:34.301465 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:34.301432 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:34.923732 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:34.923696 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:35.916724 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:35.916693 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:36.092505 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:36.092473 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:37.614667 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:37.614634 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:37.796879 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:37.796823 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:40.611265 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:40.611232 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:44.658155 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:44.658118 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 09:36:44.658647 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:44.658621 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="alertmanager" containerID="cri-o://40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7" gracePeriod=120 Nov 02 09:36:44.658735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:44.658658 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy-metric" containerID="cri-o://5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74" gracePeriod=120 Nov 02 09:36:44.658735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:44.658690 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="config-reloader" containerID="cri-o://39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea" gracePeriod=120 Nov 02 09:36:44.658845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:44.658723 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="prom-label-proxy" containerID="cri-o://8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b" gracePeriod=120 Nov 02 09:36:44.658845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:44.658656 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy-web" containerID="cri-o://973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5" gracePeriod=120 Nov 02 09:36:44.658845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:44.658706 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy" containerID="cri-o://54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36" gracePeriod=120 Nov 02 09:36:45.764350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764319 4467 generic.go:356] "Generic (PLEG): container finished" podID="86fc3914-4c5e-4826-9484-0c84c891377e" containerID="8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b" exitCode=0 Nov 02 09:36:45.764350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764345 4467 generic.go:356] "Generic (PLEG): container finished" podID="86fc3914-4c5e-4826-9484-0c84c891377e" containerID="5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74" exitCode=0 Nov 02 09:36:45.764350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764352 4467 generic.go:356] "Generic (PLEG): container finished" podID="86fc3914-4c5e-4826-9484-0c84c891377e" containerID="54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36" exitCode=0 Nov 02 09:36:45.764350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764358 4467 generic.go:356] "Generic (PLEG): container finished" podID="86fc3914-4c5e-4826-9484-0c84c891377e" containerID="39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea" exitCode=0 Nov 02 09:36:45.764802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764364 4467 generic.go:356] "Generic (PLEG): container finished" podID="86fc3914-4c5e-4826-9484-0c84c891377e" containerID="40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7" exitCode=0 Nov 02 09:36:45.764802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764390 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerDied","Data":"8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b"} Nov 02 09:36:45.764802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764455 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerDied","Data":"5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74"} Nov 02 09:36:45.764802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764481 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerDied","Data":"54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36"} Nov 02 09:36:45.764802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764505 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerDied","Data":"39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea"} Nov 02 09:36:45.764802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.764529 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerDied","Data":"40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7"} Nov 02 09:36:45.905411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.905381 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:36:45.960406 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.960367 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:45.997033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.996945 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-metrics-client-ca\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.996983 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997020 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-config-volume\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-config-volume\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997049 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-main-tls\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997102 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-config-out\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997121 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-cluster-tls-config\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997143 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-web\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997182 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-main-db\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997200 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-kube-api-access-p7mxp\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-kube-api-access-p7mxp\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997615 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997377 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-metrics-client-ca" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:45.997615 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997529 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-main-db" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:36:45.997715 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997648 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-tls-assets\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-tls-assets\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997715 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997695 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-web-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-web-config\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997829 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997743 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997829 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997787 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy\") pod \"86fc3914-4c5e-4826-9484-0c84c891377e\" (UID: \"86fc3914-4c5e-4826-9484-0c84c891377e\") " Nov 02 09:36:45.997968 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.997865 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:45.998106 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.998085 4467 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-main-db\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:45.998182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.998113 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-metrics-client-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:45.998182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:45.998130 4467 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fc3914-4c5e-4826-9484-0c84c891377e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.000318 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.000267 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:46.001024 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.000980 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-main-tls" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:46.001835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.001773 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-config-out" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:36:46.001835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.001794 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-web" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:46.002099 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.002074 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-config-volume" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:46.002210 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.002073 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-kube-api-access-p7mxp" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "kube-api-access-p7mxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:36:46.002210 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.002124 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-tls-assets" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:36:46.002592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.002568 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-metric" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:46.004814 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.004784 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-cluster-tls-config" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:46.012295 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.012269 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-web-config" pod "86fc3914-4c5e-4826-9484-0c84c891377e" (UID: "86fc3914-4c5e-4826-9484-0c84c891377e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:46.098853 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098815 4467 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86fc3914-4c5e-4826-9484-0c84c891377e-config-out\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.098853 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098846 4467 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-cluster-tls-config\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.098853 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098857 4467 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.099128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098866 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7mxp\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-kube-api-access-p7mxp\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.099128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098876 4467 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86fc3914-4c5e-4826-9484-0c84c891377e-tls-assets\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.099128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098923 4467 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-web-config\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.099128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098938 4467 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.099128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098951 4467 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.099128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098965 4467 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.099128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.098977 4467 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86fc3914-4c5e-4826-9484-0c84c891377e-secret-alertmanager-main-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:46.140081 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.140046 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:46.771288 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.771256 4467 generic.go:356] "Generic (PLEG): container finished" podID="86fc3914-4c5e-4826-9484-0c84c891377e" containerID="973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5" exitCode=0 Nov 02 09:36:46.771712 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.771319 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerDied","Data":"973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5"} Nov 02 09:36:46.771712 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.771332 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 09:36:46.771712 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.771373 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86fc3914-4c5e-4826-9484-0c84c891377e","Type":"ContainerDied","Data":"05d1d73996a289b362cb1e81d785d06f3bb7dfd69c17e7e534c713ad01d30646"} Nov 02 09:36:46.771712 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.771417 4467 scope.go:122] "RemoveContainer" containerID="8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b" Nov 02 09:36:46.779602 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.779532 4467 scope.go:122] "RemoveContainer" containerID="5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74" Nov 02 09:36:46.786884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.786865 4467 scope.go:122] "RemoveContainer" containerID="54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36" Nov 02 09:36:46.793845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.793823 4467 scope.go:122] "RemoveContainer" containerID="973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5" Nov 02 09:36:46.795454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.795431 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 09:36:46.799881 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.799859 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 09:36:46.802877 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.802859 4467 scope.go:122] "RemoveContainer" containerID="39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea" Nov 02 09:36:46.810022 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.810003 4467 scope.go:122] "RemoveContainer" containerID="40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7" Nov 02 09:36:46.817277 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.817255 4467 scope.go:122] "RemoveContainer" containerID="52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81" Nov 02 09:36:46.825247 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.825224 4467 scope.go:122] "RemoveContainer" containerID="8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b" Nov 02 09:36:46.825545 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:46.825522 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b\": container with ID starting with 8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b not found: ID does not exist" containerID="8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b" Nov 02 09:36:46.825625 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.825553 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b"} err="failed to get container status \"8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b\": rpc error: code = NotFound desc = could not find container \"8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b\": container with ID starting with 8cf29b5299068046c435db6b3642bf782fe3a27341e5da7a9536a368d33f457b not found: ID does not exist" Nov 02 09:36:46.825625 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.825590 4467 scope.go:122] "RemoveContainer" containerID="5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74" Nov 02 09:36:46.825869 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:46.825851 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74\": container with ID starting with 5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74 not found: ID does not exist" containerID="5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74" Nov 02 09:36:46.825944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.825875 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74"} err="failed to get container status \"5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74\": rpc error: code = NotFound desc = could not find container \"5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74\": container with ID starting with 5bdbd97e0481b135c7b0aa7e90ee72eb53b881a49e407231df725df1f0956b74 not found: ID does not exist" Nov 02 09:36:46.825944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.825913 4467 scope.go:122] "RemoveContainer" containerID="54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36" Nov 02 09:36:46.826152 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:46.826133 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36\": container with ID starting with 54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36 not found: ID does not exist" containerID="54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36" Nov 02 09:36:46.826199 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.826158 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36"} err="failed to get container status \"54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36\": rpc error: code = NotFound desc = could not find container \"54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36\": container with ID starting with 54479fcc075d2bda5ed7088afe47d972a78c4cf8b5b3b620162bce0b32ba5c36 not found: ID does not exist" Nov 02 09:36:46.826199 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.826173 4467 scope.go:122] "RemoveContainer" containerID="973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5" Nov 02 09:36:46.826430 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:46.826415 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5\": container with ID starting with 973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5 not found: ID does not exist" containerID="973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5" Nov 02 09:36:46.826465 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.826433 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5"} err="failed to get container status \"973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5\": rpc error: code = NotFound desc = could not find container \"973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5\": container with ID starting with 973940e7e04b7ac0b5059052bd3410dab169a7b7cf3ede2f9139fa24138685a5 not found: ID does not exist" Nov 02 09:36:46.826465 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.826445 4467 scope.go:122] "RemoveContainer" containerID="39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea" Nov 02 09:36:46.826669 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:46.826652 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea\": container with ID starting with 39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea not found: ID does not exist" containerID="39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea" Nov 02 09:36:46.826716 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.826672 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea"} err="failed to get container status \"39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea\": rpc error: code = NotFound desc = could not find container \"39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea\": container with ID starting with 39280cc83b0ea9ca122cf3ff8f04a59af14e88bd6094458ce9fd1b5748ff69ea not found: ID does not exist" Nov 02 09:36:46.826716 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.826684 4467 scope.go:122] "RemoveContainer" containerID="40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7" Nov 02 09:36:46.826922 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:46.826864 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7\": container with ID starting with 40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7 not found: ID does not exist" containerID="40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7" Nov 02 09:36:46.826922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.826881 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7"} err="failed to get container status \"40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7\": rpc error: code = NotFound desc = could not find container \"40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7\": container with ID starting with 40812be8e17455ea1beeedebde74dff2a925e7d36ca437ac91ae796a9a043be7 not found: ID does not exist" Nov 02 09:36:46.826922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.826906 4467 scope.go:122] "RemoveContainer" containerID="52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81" Nov 02 09:36:46.827132 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:46.827117 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81\": container with ID starting with 52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81 not found: ID does not exist" containerID="52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81" Nov 02 09:36:46.827170 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:46.827134 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81"} err="failed to get container status \"52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81\": rpc error: code = NotFound desc = could not find container \"52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81\": container with ID starting with 52fddd73fe57310639e5b9e3c75e9d29d041f35d009e1794e662a9c310edff81 not found: ID does not exist" Nov 02 09:36:47.329667 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:47.329625 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" path="/var/lib/kubelet/pods/86fc3914-4c5e-4826-9484-0c84c891377e/volumes" Nov 02 09:36:48.967107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:48.967062 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:36:48.967508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:48.967384 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="prometheus" containerID="cri-o://ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" gracePeriod=600 Nov 02 09:36:48.967745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:48.967704 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" gracePeriod=600 Nov 02 09:36:48.967817 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:48.967803 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy" containerID="cri-o://52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" gracePeriod=600 Nov 02 09:36:48.967906 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:48.967856 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy-web" containerID="cri-o://4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" gracePeriod=600 Nov 02 09:36:48.967980 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:48.967942 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="thanos-sidecar" containerID="cri-o://27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" gracePeriod=600 Nov 02 09:36:48.968035 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:48.968005 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="config-reloader" containerID="cri-o://03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" gracePeriod=600 Nov 02 09:36:49.219595 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.218572 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.327563 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327528 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-config\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.327757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327577 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-1\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.327757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327606 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-thanos-prometheus-http-client-file\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.327757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327646 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-kube-rbac-proxy\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.327757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327677 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-metrics-client-certs\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.327757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327725 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-web-config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-web-config\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327761 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-kube-api-access-c6ghc\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-kube-api-access-c6ghc\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327789 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-grpc-tls\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327824 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-2\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327859 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-db\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327903 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-metrics-client-ca\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327930 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-tls\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.327986 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-config-out\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.328014 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-tls-assets\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-tls-assets\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.328040 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.328074 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-0\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.328123 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-serving-certs-ca-bundle\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.328173 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-kubelet-serving-ca-bundle\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.328206 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-trusted-ca-bundle\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.328403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.328241 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5193e406-14fa-46c0-adf6-63c3061817e2\" (UID: \"5193e406-14fa-46c0-adf6-63c3061817e2\") " Nov 02 09:36:49.329428 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.329330 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-serving-certs-ca-bundle" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:49.329723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.329571 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-db" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:36:49.330858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.330833 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-0" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:49.331066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.331038 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-thanos-sidecar-tls" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.331884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.331644 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-config-out" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:36:49.332480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.332053 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-kube-rbac-proxy-web" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.332480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.332194 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-trusted-ca-bundle" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:49.332623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.332484 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-kubelet-serving-ca-bundle" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:49.332795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.332770 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-metrics-client-ca" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:49.332871 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.332846 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-1" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:49.333379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.333349 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-thanos-prometheus-http-client-file" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.333379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.333371 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-metrics-client-certs" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.333534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.333417 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-tls-assets" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:36:49.333673 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.333648 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-config" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.333763 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.333731 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-2" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:36:49.334515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.334489 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-tls" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.335152 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.335124 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-grpc-tls" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.335312 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.335298 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-kube-rbac-proxy" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.336364 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.336346 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-kube-api-access-c6ghc" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "kube-api-access-c6ghc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:36:49.345020 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.344997 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-web-config" pod "5193e406-14fa-46c0-adf6-63c3061817e2" (UID: "5193e406-14fa-46c0-adf6-63c3061817e2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:36:49.429503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429469 4467 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429496 4467 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429508 4467 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429519 4467 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429530 4467 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-config\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429541 4467 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429552 4467 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429561 4467 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-kube-rbac-proxy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429570 4467 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-metrics-client-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429600 4467 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-web-config\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429609 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6ghc\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-kube-api-access-c6ghc\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429617 4467 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-grpc-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429625 4467 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429634 4467 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-db\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429642 4467 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-configmap-metrics-client-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429650 4467 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429659 4467 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5193e406-14fa-46c0-adf6-63c3061817e2-config-out\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429668 4467 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5193e406-14fa-46c0-adf6-63c3061817e2-tls-assets\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429676 4467 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5193e406-14fa-46c0-adf6-63c3061817e2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.429727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.429685 4467 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5193e406-14fa-46c0-adf6-63c3061817e2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:36:49.800750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800714 4467 generic.go:356] "Generic (PLEG): container finished" podID="5193e406-14fa-46c0-adf6-63c3061817e2" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" exitCode=0 Nov 02 09:36:49.800750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800739 4467 generic.go:356] "Generic (PLEG): container finished" podID="5193e406-14fa-46c0-adf6-63c3061817e2" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" exitCode=0 Nov 02 09:36:49.800750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800746 4467 generic.go:356] "Generic (PLEG): container finished" podID="5193e406-14fa-46c0-adf6-63c3061817e2" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" exitCode=0 Nov 02 09:36:49.800750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800752 4467 generic.go:356] "Generic (PLEG): container finished" podID="5193e406-14fa-46c0-adf6-63c3061817e2" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" exitCode=0 Nov 02 09:36:49.800750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800757 4467 generic.go:356] "Generic (PLEG): container finished" podID="5193e406-14fa-46c0-adf6-63c3061817e2" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" exitCode=0 Nov 02 09:36:49.800750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800762 4467 generic.go:356] "Generic (PLEG): container finished" podID="5193e406-14fa-46c0-adf6-63c3061817e2" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" exitCode=0 Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800788 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerDied","Data":"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e"} Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800804 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800854 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerDied","Data":"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3"} Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800881 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerDied","Data":"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26"} Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800920 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerDied","Data":"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e"} Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800926 4467 scope.go:122] "RemoveContainer" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800945 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerDied","Data":"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d"} Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800968 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerDied","Data":"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8"} Nov 02 09:36:49.801219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.800992 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5193e406-14fa-46c0-adf6-63c3061817e2","Type":"ContainerDied","Data":"ab42df41690bd6afd3cefc12357b6ccab84d4bc18b7e5fad6dfe67d52baecb5d"} Nov 02 09:36:49.808417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.808399 4467 scope.go:122] "RemoveContainer" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" Nov 02 09:36:49.815107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.815091 4467 scope.go:122] "RemoveContainer" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" Nov 02 09:36:49.821488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.821465 4467 scope.go:122] "RemoveContainer" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" Nov 02 09:36:49.826015 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.825881 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:36:49.828451 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.828431 4467 scope.go:122] "RemoveContainer" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" Nov 02 09:36:49.831010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.830986 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:36:49.835931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.835911 4467 scope.go:122] "RemoveContainer" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" Nov 02 09:36:49.844911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.844858 4467 scope.go:122] "RemoveContainer" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" Nov 02 09:36:49.851835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.851813 4467 scope.go:122] "RemoveContainer" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" Nov 02 09:36:49.852368 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:49.852345 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": container with ID starting with 3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e not found: ID does not exist" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" Nov 02 09:36:49.852454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.852374 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e"} err="failed to get container status \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": rpc error: code = NotFound desc = could not find container \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": container with ID starting with 3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e not found: ID does not exist" Nov 02 09:36:49.852454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.852393 4467 scope.go:122] "RemoveContainer" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" Nov 02 09:36:49.852683 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:49.852666 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": container with ID starting with 52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3 not found: ID does not exist" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" Nov 02 09:36:49.852727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.852692 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3"} err="failed to get container status \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": rpc error: code = NotFound desc = could not find container \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": container with ID starting with 52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3 not found: ID does not exist" Nov 02 09:36:49.852727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.852706 4467 scope.go:122] "RemoveContainer" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" Nov 02 09:36:49.852969 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:49.852949 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": container with ID starting with 4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26 not found: ID does not exist" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" Nov 02 09:36:49.853019 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.852979 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26"} err="failed to get container status \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": rpc error: code = NotFound desc = could not find container \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": container with ID starting with 4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26 not found: ID does not exist" Nov 02 09:36:49.853019 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.852998 4467 scope.go:122] "RemoveContainer" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" Nov 02 09:36:49.853287 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:49.853268 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": container with ID starting with 27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e not found: ID does not exist" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" Nov 02 09:36:49.853353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.853295 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e"} err="failed to get container status \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": rpc error: code = NotFound desc = could not find container \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": container with ID starting with 27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e not found: ID does not exist" Nov 02 09:36:49.853353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.853317 4467 scope.go:122] "RemoveContainer" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" Nov 02 09:36:49.853535 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:49.853519 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": container with ID starting with 03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d not found: ID does not exist" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" Nov 02 09:36:49.853575 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.853537 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d"} err="failed to get container status \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": rpc error: code = NotFound desc = could not find container \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": container with ID starting with 03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d not found: ID does not exist" Nov 02 09:36:49.853575 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.853550 4467 scope.go:122] "RemoveContainer" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" Nov 02 09:36:49.853752 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:49.853737 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": container with ID starting with ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8 not found: ID does not exist" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" Nov 02 09:36:49.853793 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.853755 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8"} err="failed to get container status \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": rpc error: code = NotFound desc = could not find container \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": container with ID starting with ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8 not found: ID does not exist" Nov 02 09:36:49.853793 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.853769 4467 scope.go:122] "RemoveContainer" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" Nov 02 09:36:49.853990 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:49.853976 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": container with ID starting with e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446 not found: ID does not exist" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" Nov 02 09:36:49.854036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.853993 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446"} err="failed to get container status \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": rpc error: code = NotFound desc = could not find container \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": container with ID starting with e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446 not found: ID does not exist" Nov 02 09:36:49.854036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854004 4467 scope.go:122] "RemoveContainer" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" Nov 02 09:36:49.854201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854182 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e"} err="failed to get container status \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": rpc error: code = NotFound desc = could not find container \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": container with ID starting with 3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e not found: ID does not exist" Nov 02 09:36:49.854201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854203 4467 scope.go:122] "RemoveContainer" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" Nov 02 09:36:49.854449 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854430 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3"} err="failed to get container status \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": rpc error: code = NotFound desc = could not find container \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": container with ID starting with 52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3 not found: ID does not exist" Nov 02 09:36:49.854518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854451 4467 scope.go:122] "RemoveContainer" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" Nov 02 09:36:49.854686 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854669 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26"} err="failed to get container status \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": rpc error: code = NotFound desc = could not find container \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": container with ID starting with 4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26 not found: ID does not exist" Nov 02 09:36:49.854731 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854687 4467 scope.go:122] "RemoveContainer" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" Nov 02 09:36:49.854905 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854876 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e"} err="failed to get container status \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": rpc error: code = NotFound desc = could not find container \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": container with ID starting with 27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e not found: ID does not exist" Nov 02 09:36:49.854980 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.854907 4467 scope.go:122] "RemoveContainer" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" Nov 02 09:36:49.855159 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855142 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d"} err="failed to get container status \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": rpc error: code = NotFound desc = could not find container \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": container with ID starting with 03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d not found: ID does not exist" Nov 02 09:36:49.855207 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855160 4467 scope.go:122] "RemoveContainer" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" Nov 02 09:36:49.855369 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855351 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8"} err="failed to get container status \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": rpc error: code = NotFound desc = could not find container \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": container with ID starting with ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8 not found: ID does not exist" Nov 02 09:36:49.855408 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855368 4467 scope.go:122] "RemoveContainer" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" Nov 02 09:36:49.855566 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855551 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446"} err="failed to get container status \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": rpc error: code = NotFound desc = could not find container \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": container with ID starting with e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446 not found: ID does not exist" Nov 02 09:36:49.855609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855566 4467 scope.go:122] "RemoveContainer" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" Nov 02 09:36:49.855754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855738 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e"} err="failed to get container status \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": rpc error: code = NotFound desc = could not find container \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": container with ID starting with 3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e not found: ID does not exist" Nov 02 09:36:49.855791 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855754 4467 scope.go:122] "RemoveContainer" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" Nov 02 09:36:49.855952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855936 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3"} err="failed to get container status \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": rpc error: code = NotFound desc = could not find container \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": container with ID starting with 52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3 not found: ID does not exist" Nov 02 09:36:49.855992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.855952 4467 scope.go:122] "RemoveContainer" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" Nov 02 09:36:49.856136 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856120 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26"} err="failed to get container status \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": rpc error: code = NotFound desc = could not find container \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": container with ID starting with 4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26 not found: ID does not exist" Nov 02 09:36:49.856174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856136 4467 scope.go:122] "RemoveContainer" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" Nov 02 09:36:49.856338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856323 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e"} err="failed to get container status \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": rpc error: code = NotFound desc = could not find container \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": container with ID starting with 27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e not found: ID does not exist" Nov 02 09:36:49.856375 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856337 4467 scope.go:122] "RemoveContainer" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" Nov 02 09:36:49.856519 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856504 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d"} err="failed to get container status \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": rpc error: code = NotFound desc = could not find container \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": container with ID starting with 03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d not found: ID does not exist" Nov 02 09:36:49.856565 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856519 4467 scope.go:122] "RemoveContainer" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" Nov 02 09:36:49.856700 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856684 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8"} err="failed to get container status \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": rpc error: code = NotFound desc = could not find container \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": container with ID starting with ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8 not found: ID does not exist" Nov 02 09:36:49.856700 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856698 4467 scope.go:122] "RemoveContainer" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" Nov 02 09:36:49.856876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856861 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446"} err="failed to get container status \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": rpc error: code = NotFound desc = could not find container \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": container with ID starting with e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446 not found: ID does not exist" Nov 02 09:36:49.856938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.856875 4467 scope.go:122] "RemoveContainer" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" Nov 02 09:36:49.857075 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857060 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e"} err="failed to get container status \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": rpc error: code = NotFound desc = could not find container \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": container with ID starting with 3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e not found: ID does not exist" Nov 02 09:36:49.857124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857075 4467 scope.go:122] "RemoveContainer" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" Nov 02 09:36:49.857306 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857284 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3"} err="failed to get container status \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": rpc error: code = NotFound desc = could not find container \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": container with ID starting with 52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3 not found: ID does not exist" Nov 02 09:36:49.857306 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857303 4467 scope.go:122] "RemoveContainer" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" Nov 02 09:36:49.857548 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857533 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26"} err="failed to get container status \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": rpc error: code = NotFound desc = could not find container \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": container with ID starting with 4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26 not found: ID does not exist" Nov 02 09:36:49.857548 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857548 4467 scope.go:122] "RemoveContainer" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" Nov 02 09:36:49.857759 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857744 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e"} err="failed to get container status \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": rpc error: code = NotFound desc = could not find container \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": container with ID starting with 27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e not found: ID does not exist" Nov 02 09:36:49.857810 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857759 4467 scope.go:122] "RemoveContainer" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" Nov 02 09:36:49.857975 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857959 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d"} err="failed to get container status \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": rpc error: code = NotFound desc = could not find container \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": container with ID starting with 03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d not found: ID does not exist" Nov 02 09:36:49.858019 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.857975 4467 scope.go:122] "RemoveContainer" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" Nov 02 09:36:49.858177 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858161 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8"} err="failed to get container status \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": rpc error: code = NotFound desc = could not find container \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": container with ID starting with ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8 not found: ID does not exist" Nov 02 09:36:49.858214 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858178 4467 scope.go:122] "RemoveContainer" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" Nov 02 09:36:49.858378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858362 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446"} err="failed to get container status \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": rpc error: code = NotFound desc = could not find container \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": container with ID starting with e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446 not found: ID does not exist" Nov 02 09:36:49.858426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858378 4467 scope.go:122] "RemoveContainer" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" Nov 02 09:36:49.858560 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858544 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e"} err="failed to get container status \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": rpc error: code = NotFound desc = could not find container \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": container with ID starting with 3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e not found: ID does not exist" Nov 02 09:36:49.858599 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858560 4467 scope.go:122] "RemoveContainer" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" Nov 02 09:36:49.858752 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858737 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3"} err="failed to get container status \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": rpc error: code = NotFound desc = could not find container \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": container with ID starting with 52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3 not found: ID does not exist" Nov 02 09:36:49.858794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858752 4467 scope.go:122] "RemoveContainer" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" Nov 02 09:36:49.858972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858944 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26"} err="failed to get container status \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": rpc error: code = NotFound desc = could not find container \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": container with ID starting with 4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26 not found: ID does not exist" Nov 02 09:36:49.858972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.858971 4467 scope.go:122] "RemoveContainer" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" Nov 02 09:36:49.859200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.859180 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e"} err="failed to get container status \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": rpc error: code = NotFound desc = could not find container \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": container with ID starting with 27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e not found: ID does not exist" Nov 02 09:36:49.859200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.859199 4467 scope.go:122] "RemoveContainer" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" Nov 02 09:36:49.859667 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.859561 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d"} err="failed to get container status \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": rpc error: code = NotFound desc = could not find container \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": container with ID starting with 03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d not found: ID does not exist" Nov 02 09:36:49.859667 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.859591 4467 scope.go:122] "RemoveContainer" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" Nov 02 09:36:49.860198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.860168 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8"} err="failed to get container status \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": rpc error: code = NotFound desc = could not find container \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": container with ID starting with ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8 not found: ID does not exist" Nov 02 09:36:49.860198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.860191 4467 scope.go:122] "RemoveContainer" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" Nov 02 09:36:49.860473 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.860453 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446"} err="failed to get container status \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": rpc error: code = NotFound desc = could not find container \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": container with ID starting with e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446 not found: ID does not exist" Nov 02 09:36:49.860546 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.860475 4467 scope.go:122] "RemoveContainer" containerID="3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e" Nov 02 09:36:49.860713 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.860691 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e"} err="failed to get container status \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": rpc error: code = NotFound desc = could not find container \"3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e\": container with ID starting with 3ac641f8e8bce92103dcdc5230ca8b7f6179b5a732a7b1ccc36713a1076d578e not found: ID does not exist" Nov 02 09:36:49.860768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.860714 4467 scope.go:122] "RemoveContainer" containerID="52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3" Nov 02 09:36:49.861025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861005 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3"} err="failed to get container status \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": rpc error: code = NotFound desc = could not find container \"52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3\": container with ID starting with 52aca82342790d35e2e0341f9d2181a5240ea9776e8bd87083d05af7443db2f3 not found: ID does not exist" Nov 02 09:36:49.861107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861026 4467 scope.go:122] "RemoveContainer" containerID="4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26" Nov 02 09:36:49.861313 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861297 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:36:49.861347 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861306 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26"} err="failed to get container status \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": rpc error: code = NotFound desc = could not find container \"4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26\": container with ID starting with 4112c265f55fb0b0378dc3319022029936057df261deab0501a790bb20b89c26 not found: ID does not exist" Nov 02 09:36:49.861347 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861322 4467 scope.go:122] "RemoveContainer" containerID="27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e" Nov 02 09:36:49.861548 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861530 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e"} err="failed to get container status \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": rpc error: code = NotFound desc = could not find container \"27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e\": container with ID starting with 27256eaf53538a0d5214c5c010bca3b36196c91d1f983b0e317883f1f2a6313e not found: ID does not exist" Nov 02 09:36:49.861611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861549 4467 scope.go:122] "RemoveContainer" containerID="03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d" Nov 02 09:36:49.861611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861568 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="prometheus" Nov 02 09:36:49.861611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861580 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="prometheus" Nov 02 09:36:49.861611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861588 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="config-reloader" Nov 02 09:36:49.861611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861593 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="config-reloader" Nov 02 09:36:49.861611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861601 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy-thanos" Nov 02 09:36:49.861611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861608 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy-thanos" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861619 4467 cpu_manager.go:414] "containerMap: removing container" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="config-reloader" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861626 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="config-reloader" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861635 4467 cpu_manager.go:414] "containerMap: removing container" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="prom-label-proxy" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861642 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="prom-label-proxy" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861656 4467 cpu_manager.go:414] "containerMap: removing container" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="init-config-reloader" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861663 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="init-config-reloader" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861677 4467 cpu_manager.go:414] "containerMap: removing container" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy-web" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861684 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy-web" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861693 4467 cpu_manager.go:414] "containerMap: removing container" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy-metric" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861700 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy-metric" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861714 4467 cpu_manager.go:414] "containerMap: removing container" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861722 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861731 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861738 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861747 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="init-config-reloader" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861752 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="init-config-reloader" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861757 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy-web" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861762 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy-web" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861771 4467 cpu_manager.go:414] "containerMap: removing container" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="alertmanager" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861775 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="alertmanager" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861782 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="thanos-sidecar" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861787 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="thanos-sidecar" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861787 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d"} err="failed to get container status \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": rpc error: code = NotFound desc = could not find container \"03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d\": container with ID starting with 03d777a3896d66601560ed500abe547a25d0a7251f4341db3deb7d8c127a4e8d not found: ID does not exist" Nov 02 09:36:49.861831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861802 4467 scope.go:122] "RemoveContainer" containerID="ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861848 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="prometheus" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861856 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="config-reloader" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861862 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy-thanos" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861912 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="config-reloader" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861919 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="prom-label-proxy" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861926 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy-web" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861931 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy-metric" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861940 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="kube-rbac-proxy" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861949 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861957 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="kube-rbac-proxy-web" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861966 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc3914-4c5e-4826-9484-0c84c891377e" containerName="alertmanager" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.861973 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" containerName="thanos-sidecar" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.862036 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8"} err="failed to get container status \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": rpc error: code = NotFound desc = could not find container \"ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8\": container with ID starting with ac23aaa327bb49334082cd7d8dd02e27d43faaa59ae84c09ca9f0d097640c0b8 not found: ID does not exist" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.862048 4467 scope.go:122] "RemoveContainer" containerID="e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446" Nov 02 09:36:49.862674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.862272 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446"} err="failed to get container status \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": rpc error: code = NotFound desc = could not find container \"e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446\": container with ID starting with e8646e77d8d21e6c34736f6fba163e549ece121735462d21b6545f193c99e446 not found: ID does not exist" Nov 02 09:36:49.867295 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.867278 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.872087 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872067 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 09:36:49.872253 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872234 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 09:36:49.872370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872236 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 09:36:49.872370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872237 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 09:36:49.872370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872244 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 09:36:49.872370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872302 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 09:36:49.872615 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872375 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 09:36:49.872615 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872405 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 09:36:49.872615 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872426 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 09:36:49.872615 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872429 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bjq15mnn2s3mt\"" Nov 02 09:36:49.872748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872621 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 09:36:49.872748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872627 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 09:36:49.872748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.872636 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 09:36:49.875425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.875409 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dvlfc\"" Nov 02 09:36:49.877214 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.877190 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 09:36:49.877309 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.877297 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 09:36:49.884163 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.884138 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:36:49.933301 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933257 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933301 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933293 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933312 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933377 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933414 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ed1e880-0148-4f30-827f-5da46b91c2cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933475 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933496 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933515 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933556 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933601 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933624 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933641 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-config\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933656 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933671 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wv9p\" (UniqueName: \"kubernetes.io/projected/6ed1e880-0148-4f30-827f-5da46b91c2cf-kube-api-access-7wv9p\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.933748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933696 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ed1e880-0148-4f30-827f-5da46b91c2cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.934020 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933778 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.934020 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933807 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.934020 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933857 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.934020 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933878 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:49.934020 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:49.933932 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.035174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035136 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.035174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035175 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.035704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035195 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.035704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035313 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.035704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035361 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.035704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035393 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.035704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035428 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-config\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.035704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035539 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035928 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035969 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wv9p\" (UniqueName: \"kubernetes.io/projected/6ed1e880-0148-4f30-827f-5da46b91c2cf-kube-api-access-7wv9p\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.035991 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036004 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ed1e880-0148-4f30-827f-5da46b91c2cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036048 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036073 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036107 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036150 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036186 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036222 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036255 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036287 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036311 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.036745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.036349 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ed1e880-0148-4f30-827f-5da46b91c2cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.038472 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.038441 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-config\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.038650 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.038627 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.038724 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.038668 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.038724 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.038671 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ed1e880-0148-4f30-827f-5da46b91c2cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.038980 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.038955 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.039719 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.039119 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ed1e880-0148-4f30-827f-5da46b91c2cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.039719 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.039361 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.039719 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.039489 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.039719 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.039523 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.040006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.039807 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.040188 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.040166 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.040188 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.040179 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ed1e880-0148-4f30-827f-5da46b91c2cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.040497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.040480 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.041131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.041110 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.041341 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.041321 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.041587 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.041573 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.042354 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.042333 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ed1e880-0148-4f30-827f-5da46b91c2cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.048468 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.048448 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wv9p\" (UniqueName: \"kubernetes.io/projected/6ed1e880-0148-4f30-827f-5da46b91c2cf-kube-api-access-7wv9p\") pod \"prometheus-k8s-0\" (UID: \"6ed1e880-0148-4f30-827f-5da46b91c2cf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.196200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.196169 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:50.482594 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.482570 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 09:36:50.484435 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:36:50.484410 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed1e880_0148_4f30_827f_5da46b91c2cf.slice/crio-a60e297318999fc36d95b0c0c669456dc1417f2911ec761df5a08b71833377ba WatchSource:0}: Error finding container a60e297318999fc36d95b0c0c669456dc1417f2911ec761df5a08b71833377ba: Status 404 returned error can't find the container with id a60e297318999fc36d95b0c0c669456dc1417f2911ec761df5a08b71833377ba Nov 02 09:36:50.806458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.806369 4467 generic.go:356] "Generic (PLEG): container finished" podID="6ed1e880-0148-4f30-827f-5da46b91c2cf" containerID="947801bd0b7666686401d6e2fa22e9ae0a0d97e4d4e29c5f5b609ee469db232c" exitCode=0 Nov 02 09:36:50.806458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.806442 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ed1e880-0148-4f30-827f-5da46b91c2cf","Type":"ContainerDied","Data":"947801bd0b7666686401d6e2fa22e9ae0a0d97e4d4e29c5f5b609ee469db232c"} Nov 02 09:36:50.806688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:50.806517 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ed1e880-0148-4f30-827f-5da46b91c2cf","Type":"ContainerStarted","Data":"a60e297318999fc36d95b0c0c669456dc1417f2911ec761df5a08b71833377ba"} Nov 02 09:36:51.329714 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:51.329638 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5193e406-14fa-46c0-adf6-63c3061817e2" path="/var/lib/kubelet/pods/5193e406-14fa-46c0-adf6-63c3061817e2/volumes" Nov 02 09:36:51.819186 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:51.819144 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ed1e880-0148-4f30-827f-5da46b91c2cf","Type":"ContainerStarted","Data":"79817fd9eb69f956782f8044722ef8c9dd9acd91bb703113148dfd2f573b3abb"} Nov 02 09:36:51.819346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:51.819240 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ed1e880-0148-4f30-827f-5da46b91c2cf","Type":"ContainerStarted","Data":"b7a0006062b476e1a3c35b7fe6eea0c2770305cba2e13ac3acf83bda810413db"} Nov 02 09:36:51.819346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:51.819269 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ed1e880-0148-4f30-827f-5da46b91c2cf","Type":"ContainerStarted","Data":"0ddb2b7624f8c97eeb07a24106ff95ae519e4bba99a7e1a38d2a753ecc5c42a4"} Nov 02 09:36:51.819346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:51.819293 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ed1e880-0148-4f30-827f-5da46b91c2cf","Type":"ContainerStarted","Data":"c586e204b20d7ca3110b5ef37931b5ce79f2e4d7be143749738ad82cdc70d595"} Nov 02 09:36:51.819346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:51.819315 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ed1e880-0148-4f30-827f-5da46b91c2cf","Type":"ContainerStarted","Data":"abc9e59730dcf7d0bcc4979caa9f962a82b4ee9e2d25954b778a490a669b221a"} Nov 02 09:36:51.819346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:51.819337 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ed1e880-0148-4f30-827f-5da46b91c2cf","Type":"ContainerStarted","Data":"373ee8c88109054ab86ca30c2e644e4951d702af5175436b8445f975c2dafee4"} Nov 02 09:36:51.854469 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:51.854403 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.837778319 podStartE2EDuration="2.8543867s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.608381ms" isStatefulPod=false podCreationTimestamp="2026-11-02 09:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:36:51.85419311 +0000 UTC m=+90.694135105" watchObservedRunningTime="2026-11-02 09:36:51.8543867 +0000 UTC m=+90.694328700" Nov 02 09:36:52.621835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:52.621798 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:52.808663 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:52.808628 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:54.104122 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:54.104085 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:54.278776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:54.278736 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:54.458339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:54.458253 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:36:55.196816 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:36:55.196773 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:36:56.852119 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:36:56.852084 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:37:10.083427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:37:10.083376 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:37:26.854073 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:37:26.854037 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:37:50.196566 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:37:50.196521 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:37:50.212236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:37:50.212205 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:37:51.044879 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:37:51.044853 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 09:37:56.852740 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:37:56.852700 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:38:10.141131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:10.141094 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:19.598132 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:19.598101 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:19.784820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:19.784785 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:26.853400 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:38:26.853363 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:38:27.287695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:27.287609 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:27.467549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:27.467519 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:35.122369 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:35.122335 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:44.147066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:44.147030 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:44.355499 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:44.355464 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:45.549297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:45.549260 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:45.764119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:45.764089 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:46.776304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:46.776263 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:50.212540 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:50.212510 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:51.382547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:51.382514 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:51.561336 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:51.561299 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:52.393913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:52.393859 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:54.388777 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:38:54.388743 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:38:56.853967 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:38:56.853924 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:39:10.179724 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:10.179646 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:10.373595 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:10.373559 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:23.700800 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:23.700767 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:23.885288 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:23.885260 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:24.419849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.419809 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:24.447360 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.447324 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt"] Nov 02 09:39:24.451296 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.451270 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.478292 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.478262 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt"] Nov 02 09:39:24.535100 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.535064 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 09:39:24.535279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.535177 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 09:39:24.535279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.535210 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 09:39:24.535379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.535287 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 09:39:24.535379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.535349 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-ptvwz\"" Nov 02 09:39:24.535448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.535424 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 09:39:24.559852 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.559822 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.559961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.559869 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3427c5a6-d489-47b8-b398-4f18f09286e7-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.559961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.559954 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.560030 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.560010 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pnt\" (UniqueName: \"kubernetes.io/projected/3427c5a6-d489-47b8-b398-4f18f09286e7-kube-api-access-g6pnt\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.560070 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.560037 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.560122 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.560094 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3427c5a6-d489-47b8-b398-4f18f09286e7-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.661065 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.661026 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.661230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.661088 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3427c5a6-d489-47b8-b398-4f18f09286e7-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.661230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.661122 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.661230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.661150 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3427c5a6-d489-47b8-b398-4f18f09286e7-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.661230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.661180 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.661230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.661209 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pnt\" (UniqueName: \"kubernetes.io/projected/3427c5a6-d489-47b8-b398-4f18f09286e7-kube-api-access-g6pnt\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.661604 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.661568 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3427c5a6-d489-47b8-b398-4f18f09286e7-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.663724 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.663695 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3427c5a6-d489-47b8-b398-4f18f09286e7-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.663837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.663765 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.663837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.663814 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.663957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.663933 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3427c5a6-d489-47b8-b398-4f18f09286e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.674770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.674712 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pnt\" (UniqueName: \"kubernetes.io/projected/3427c5a6-d489-47b8-b398-4f18f09286e7-kube-api-access-g6pnt\") pod \"opendatahub-operator-controller-manager-f7698d755-48lmt\" (UID: \"3427c5a6-d489-47b8-b398-4f18f09286e7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.677861 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.677839 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:24.767281 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.767248 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:24.869942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:24.869898 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:25.064960 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:25.064931 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt"] Nov 02 09:39:25.067398 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:39:25.067370 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3427c5a6_d489_47b8_b398_4f18f09286e7.slice/crio-c9c019557453a288f4ac2fa0602bf70cfa3bac2f57558d78385509aa2dd0d509 WatchSource:0}: Error finding container c9c019557453a288f4ac2fa0602bf70cfa3bac2f57558d78385509aa2dd0d509: Status 404 returned error can't find the container with id c9c019557453a288f4ac2fa0602bf70cfa3bac2f57558d78385509aa2dd0d509 Nov 02 09:39:25.068467 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:25.068444 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:25.340948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:25.340863 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" event={"ID":"3427c5a6-d489-47b8-b398-4f18f09286e7","Type":"ContainerStarted","Data":"c9c019557453a288f4ac2fa0602bf70cfa3bac2f57558d78385509aa2dd0d509"} Nov 02 09:39:26.856973 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:39:26.856939 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:39:27.933445 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:27.933390 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:28.353202 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:28.353167 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" event={"ID":"3427c5a6-d489-47b8-b398-4f18f09286e7","Type":"ContainerStarted","Data":"484de6cdc412a7b34f86f9061a8987a4731c0a18e3c4781dfeefce97cba384b7"} Nov 02 09:39:28.676817 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:28.676727 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:29.357629 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:29.357589 4467 generic.go:356] "Generic (PLEG): container finished" podID="3427c5a6-d489-47b8-b398-4f18f09286e7" containerID="484de6cdc412a7b34f86f9061a8987a4731c0a18e3c4781dfeefce97cba384b7" exitCode=0 Nov 02 09:39:29.358051 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:29.357677 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" event={"ID":"3427c5a6-d489-47b8-b398-4f18f09286e7","Type":"ContainerDied","Data":"484de6cdc412a7b34f86f9061a8987a4731c0a18e3c4781dfeefce97cba384b7"} Nov 02 09:39:30.324676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:30.324640 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:30.363624 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:30.363591 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" event={"ID":"3427c5a6-d489-47b8-b398-4f18f09286e7","Type":"ContainerStarted","Data":"3658653efefe01e48c8946c53ba1f87d750f6340fc6cd424ab6c597d502b10f2"} Nov 02 09:39:30.364127 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:30.363724 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:30.404542 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:30.404486 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" podStartSLOduration=2.581580839 podStartE2EDuration="6.404470712s" totalImagesPullingTime="3.396296365s" totalInitContainerRuntime="426.593508ms" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:30.402367761 +0000 UTC m=+249.242309760" watchObservedRunningTime="2026-11-02 09:39:30.404470712 +0000 UTC m=+249.244412707" Nov 02 09:39:30.511363 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:30.511309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:31.721235 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:31.721195 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:31.929191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:31.929143 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:33.328451 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:33.328409 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:34.082262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.082209 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:34.862604 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.862568 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-df694"] Nov 02 09:39:34.867767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.867749 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" Nov 02 09:39:34.876112 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.876086 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-9mdcz\"" Nov 02 09:39:34.876279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.876126 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 09:39:34.876279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.876244 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 09:39:34.876479 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.876460 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-df694"] Nov 02 09:39:34.950371 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.950334 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlw9\" (UniqueName: \"kubernetes.io/projected/17892573-e467-4794-ae0a-937c6c472028-kube-api-access-qrlw9\") pod \"jobset-operator-674cd59dc5-df694\" (UID: \"17892573-e467-4794-ae0a-937c6c472028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" Nov 02 09:39:34.950540 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:34.950446 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17892573-e467-4794-ae0a-937c6c472028-tmp\") pod \"jobset-operator-674cd59dc5-df694\" (UID: \"17892573-e467-4794-ae0a-937c6c472028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" Nov 02 09:39:35.050935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:35.050856 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlw9\" (UniqueName: \"kubernetes.io/projected/17892573-e467-4794-ae0a-937c6c472028-kube-api-access-qrlw9\") pod \"jobset-operator-674cd59dc5-df694\" (UID: \"17892573-e467-4794-ae0a-937c6c472028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" Nov 02 09:39:35.051140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:35.050983 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17892573-e467-4794-ae0a-937c6c472028-tmp\") pod \"jobset-operator-674cd59dc5-df694\" (UID: \"17892573-e467-4794-ae0a-937c6c472028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" Nov 02 09:39:35.051317 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:35.051301 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17892573-e467-4794-ae0a-937c6c472028-tmp\") pod \"jobset-operator-674cd59dc5-df694\" (UID: \"17892573-e467-4794-ae0a-937c6c472028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" Nov 02 09:39:35.069455 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:35.069424 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlw9\" (UniqueName: \"kubernetes.io/projected/17892573-e467-4794-ae0a-937c6c472028-kube-api-access-qrlw9\") pod \"jobset-operator-674cd59dc5-df694\" (UID: \"17892573-e467-4794-ae0a-937c6c472028\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" Nov 02 09:39:35.189414 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:35.189324 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" Nov 02 09:39:35.459808 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:35.459741 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-df694"] Nov 02 09:39:35.463114 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:39:35.463088 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17892573_e467_4794_ae0a_937c6c472028.slice/crio-0de8976d482bdd190248bc51f1ae3c8d5b262f12659848e4962700f84c9ecbb6 WatchSource:0}: Error finding container 0de8976d482bdd190248bc51f1ae3c8d5b262f12659848e4962700f84c9ecbb6: Status 404 returned error can't find the container with id 0de8976d482bdd190248bc51f1ae3c8d5b262f12659848e4962700f84c9ecbb6 Nov 02 09:39:36.386280 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:36.386242 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" event={"ID":"17892573-e467-4794-ae0a-937c6c472028","Type":"ContainerStarted","Data":"0de8976d482bdd190248bc51f1ae3c8d5b262f12659848e4962700f84c9ecbb6"} Nov 02 09:39:38.495736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:38.495702 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:39.130091 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:39.130055 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:41.370546 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:41.370519 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-48lmt" Nov 02 09:39:41.569436 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:41.569404 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:43.758194 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:43.758158 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:47.019964 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:47.019926 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:47.252204 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:47.252171 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:47.461944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:47.461911 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:48.244343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:48.244312 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:48.435227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:48.435195 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:50.645396 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.645349 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:50.818016 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.817977 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp"] Nov 02 09:39:50.821923 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.821877 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" Nov 02 09:39:50.831982 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.831466 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 09:39:50.833795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.832797 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 09:39:50.833795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.833736 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp"] Nov 02 09:39:50.856928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.856885 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:50.870127 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.870106 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2q8p7\"" Nov 02 09:39:50.883927 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.883877 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/072580f0-95f2-4082-b877-c0134f0e1a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-457dp\" (UID: \"072580f0-95f2-4082-b877-c0134f0e1a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" Nov 02 09:39:50.884084 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.883993 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmqh\" (UniqueName: \"kubernetes.io/projected/072580f0-95f2-4082-b877-c0134f0e1a26-kube-api-access-wtmqh\") pod \"cert-manager-cainjector-5c57bbd4fc-457dp\" (UID: \"072580f0-95f2-4082-b877-c0134f0e1a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" Nov 02 09:39:50.985013 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.984923 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/072580f0-95f2-4082-b877-c0134f0e1a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-457dp\" (UID: \"072580f0-95f2-4082-b877-c0134f0e1a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" Nov 02 09:39:50.985174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:50.985016 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmqh\" (UniqueName: \"kubernetes.io/projected/072580f0-95f2-4082-b877-c0134f0e1a26-kube-api-access-wtmqh\") pod \"cert-manager-cainjector-5c57bbd4fc-457dp\" (UID: \"072580f0-95f2-4082-b877-c0134f0e1a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" Nov 02 09:39:51.003434 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:51.003399 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/072580f0-95f2-4082-b877-c0134f0e1a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-457dp\" (UID: \"072580f0-95f2-4082-b877-c0134f0e1a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" Nov 02 09:39:51.004454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:51.004426 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmqh\" (UniqueName: \"kubernetes.io/projected/072580f0-95f2-4082-b877-c0134f0e1a26-kube-api-access-wtmqh\") pod \"cert-manager-cainjector-5c57bbd4fc-457dp\" (UID: \"072580f0-95f2-4082-b877-c0134f0e1a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" Nov 02 09:39:51.161281 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:51.161029 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" Nov 02 09:39:51.447230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:51.447188 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" event={"ID":"17892573-e467-4794-ae0a-937c6c472028","Type":"ContainerStarted","Data":"90a640eaba782f70e59f24b96065673d9c92946f78a8983e24288833ed6e32ce"} Nov 02 09:39:51.458462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:51.458435 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp"] Nov 02 09:39:51.461632 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:39:51.461606 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod072580f0_95f2_4082_b877_c0134f0e1a26.slice/crio-51b4bf69a766b2aa50c0b3d71dcc4e8dc7db4d0d3162cba848f2f9a0558af7a7 WatchSource:0}: Error finding container 51b4bf69a766b2aa50c0b3d71dcc4e8dc7db4d0d3162cba848f2f9a0558af7a7: Status 404 returned error can't find the container with id 51b4bf69a766b2aa50c0b3d71dcc4e8dc7db4d0d3162cba848f2f9a0558af7a7 Nov 02 09:39:51.473941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:51.473874 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-df694" podStartSLOduration=1.7717521939999998 podStartE2EDuration="17.473860632s" totalImagesPullingTime="15.702108438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:51.472803575 +0000 UTC m=+270.312745572" watchObservedRunningTime="2026-11-02 09:39:51.473860632 +0000 UTC m=+270.313802700" Nov 02 09:39:51.902286 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:51.902252 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:52.094323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:52.094288 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:52.453602 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:52.453569 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" event={"ID":"072580f0-95f2-4082-b877-c0134f0e1a26","Type":"ContainerStarted","Data":"51b4bf69a766b2aa50c0b3d71dcc4e8dc7db4d0d3162cba848f2f9a0558af7a7"} Nov 02 09:39:54.294986 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:54.294955 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:54.475241 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:54.475213 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:54.932445 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:54.932403 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:55.468762 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:55.468726 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" event={"ID":"072580f0-95f2-4082-b877-c0134f0e1a26","Type":"ContainerStarted","Data":"5d7d85e56cf8f751b39e2bf58805089ab141ec0e15b42cab491024d7a3d9ae25"} Nov 02 09:39:56.392814 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.392773 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:56.435104 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.435052 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-457dp" podStartSLOduration=2.775804657 podStartE2EDuration="6.435032539s" totalImagesPullingTime="3.659227882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:55.501592067 +0000 UTC m=+274.341534061" watchObservedRunningTime="2026-11-02 09:39:56.435032539 +0000 UTC m=+275.274974534" Nov 02 09:39:56.435276 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.435179 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw"] Nov 02 09:39:56.438438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.438422 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:56.450092 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.450073 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qkjnn\"" Nov 02 09:39:56.459521 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.459483 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw"] Nov 02 09:39:56.538028 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.535902 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkwr\" (UniqueName: \"kubernetes.io/projected/ba2ac63a-0ce6-428c-be4e-2f16c0cbb437-kube-api-access-wrkwr\") pod \"cert-manager-webhook-5dbbbf75cc-ztmqw\" (UID: \"ba2ac63a-0ce6-428c-be4e-2f16c0cbb437\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:56.538028 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.535982 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba2ac63a-0ce6-428c-be4e-2f16c0cbb437-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ztmqw\" (UID: \"ba2ac63a-0ce6-428c-be4e-2f16c0cbb437\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:56.550186 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.550155 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt"] Nov 02 09:39:56.553851 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.553821 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.562317 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.562288 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 09:39:56.562742 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.562722 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 09:39:56.563056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.563034 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 09:39:56.563056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.563053 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 09:39:56.563240 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.563166 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5cbfk\"" Nov 02 09:39:56.567612 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.567572 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt"] Nov 02 09:39:56.602264 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.602231 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:56.636598 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.636566 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkwr\" (UniqueName: \"kubernetes.io/projected/ba2ac63a-0ce6-428c-be4e-2f16c0cbb437-kube-api-access-wrkwr\") pod \"cert-manager-webhook-5dbbbf75cc-ztmqw\" (UID: \"ba2ac63a-0ce6-428c-be4e-2f16c0cbb437\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:56.636598 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.636600 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4q6z\" (UniqueName: \"kubernetes.io/projected/cb0c1710-51cc-419a-84d3-f9e57666226c-kube-api-access-c4q6z\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.636814 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.636675 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba2ac63a-0ce6-428c-be4e-2f16c0cbb437-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ztmqw\" (UID: \"ba2ac63a-0ce6-428c-be4e-2f16c0cbb437\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:56.636913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.636874 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb0c1710-51cc-419a-84d3-f9e57666226c-tls-certs\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.636960 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.636936 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb0c1710-51cc-419a-84d3-f9e57666226c-tmp\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.664289 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.664222 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba2ac63a-0ce6-428c-be4e-2f16c0cbb437-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ztmqw\" (UID: \"ba2ac63a-0ce6-428c-be4e-2f16c0cbb437\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:56.667812 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.667783 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkwr\" (UniqueName: \"kubernetes.io/projected/ba2ac63a-0ce6-428c-be4e-2f16c0cbb437-kube-api-access-wrkwr\") pod \"cert-manager-webhook-5dbbbf75cc-ztmqw\" (UID: \"ba2ac63a-0ce6-428c-be4e-2f16c0cbb437\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:56.737832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.737793 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb0c1710-51cc-419a-84d3-f9e57666226c-tls-certs\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.737832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.737837 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb0c1710-51cc-419a-84d3-f9e57666226c-tmp\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.738101 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.737873 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4q6z\" (UniqueName: \"kubernetes.io/projected/cb0c1710-51cc-419a-84d3-f9e57666226c-kube-api-access-c4q6z\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.740200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.740172 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb0c1710-51cc-419a-84d3-f9e57666226c-tmp\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.740524 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.740510 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb0c1710-51cc-419a-84d3-f9e57666226c-tls-certs\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.750306 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.750274 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:56.772994 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.772971 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4q6z\" (UniqueName: \"kubernetes.io/projected/cb0c1710-51cc-419a-84d3-f9e57666226c-kube-api-access-c4q6z\") pod \"kube-auth-proxy-7c759d848c-x8mlt\" (UID: \"cb0c1710-51cc-419a-84d3-f9e57666226c\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:56.827798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.827759 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:56.853523 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:39:56.853482 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:39:56.868776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:56.868755 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" Nov 02 09:39:57.011616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.011584 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:57.028279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.028252 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw"] Nov 02 09:39:57.030667 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:39:57.030641 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba2ac63a_0ce6_428c_be4e_2f16c0cbb437.slice/crio-aa9d900c328941026c6e11a922ca0e63e259be781dabcc775d6043e96903a6d6 WatchSource:0}: Error finding container aa9d900c328941026c6e11a922ca0e63e259be781dabcc775d6043e96903a6d6: Status 404 returned error can't find the container with id aa9d900c328941026c6e11a922ca0e63e259be781dabcc775d6043e96903a6d6 Nov 02 09:39:57.170044 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.169904 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt"] Nov 02 09:39:57.172145 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:39:57.172116 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0c1710_51cc_419a_84d3_f9e57666226c.slice/crio-f310e0a37e90a28ab9d70df3fa97d466954e1f703b692dc0e2a2b6ac24aeb736 WatchSource:0}: Error finding container f310e0a37e90a28ab9d70df3fa97d466954e1f703b692dc0e2a2b6ac24aeb736: Status 404 returned error can't find the container with id f310e0a37e90a28ab9d70df3fa97d466954e1f703b692dc0e2a2b6ac24aeb736 Nov 02 09:39:57.285770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.285740 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:57.480287 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.480187 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" event={"ID":"cb0c1710-51cc-419a-84d3-f9e57666226c","Type":"ContainerStarted","Data":"f310e0a37e90a28ab9d70df3fa97d466954e1f703b692dc0e2a2b6ac24aeb736"} Nov 02 09:39:57.481998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.481970 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" event={"ID":"ba2ac63a-0ce6-428c-be4e-2f16c0cbb437","Type":"ContainerStarted","Data":"4ab198cb7f38492987cee3609abf57f2722b005aa9819157cdb18b464a7fc687"} Nov 02 09:39:57.482138 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.482004 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" event={"ID":"ba2ac63a-0ce6-428c-be4e-2f16c0cbb437","Type":"ContainerStarted","Data":"aa9d900c328941026c6e11a922ca0e63e259be781dabcc775d6043e96903a6d6"} Nov 02 09:39:57.482138 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.482097 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:39:57.484761 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.484733 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:57.501378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.501337 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" podStartSLOduration=1.501324756 podStartE2EDuration="1.501324756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:57.500431559 +0000 UTC m=+276.340373552" watchObservedRunningTime="2026-11-02 09:39:57.501324756 +0000 UTC m=+276.341266753" Nov 02 09:39:57.660513 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.660479 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:57.901010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:57.900973 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:58.102351 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.102317 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:58.146494 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.146464 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m9jd9"] Nov 02 09:39:58.149850 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.149827 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" Nov 02 09:39:58.157840 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.157680 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 09:39:58.159679 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.158117 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 09:39:58.159679 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.159637 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m9jd9"] Nov 02 09:39:58.160944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.160925 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tnpmb\"" Nov 02 09:39:58.251213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.251178 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnvd6\" (UniqueName: \"kubernetes.io/projected/effdddfb-3a3f-4eda-8115-21a0447f7be8-kube-api-access-lnvd6\") pod \"authorino-57f7f8659f-m9jd9\" (UID: \"effdddfb-3a3f-4eda-8115-21a0447f7be8\") " pod="kuadrant-system/authorino-57f7f8659f-m9jd9" Nov 02 09:39:58.285917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.285862 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:58.352519 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.352478 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvd6\" (UniqueName: \"kubernetes.io/projected/effdddfb-3a3f-4eda-8115-21a0447f7be8-kube-api-access-lnvd6\") pod \"authorino-57f7f8659f-m9jd9\" (UID: \"effdddfb-3a3f-4eda-8115-21a0447f7be8\") " pod="kuadrant-system/authorino-57f7f8659f-m9jd9" Nov 02 09:39:58.372613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.372577 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvd6\" (UniqueName: \"kubernetes.io/projected/effdddfb-3a3f-4eda-8115-21a0447f7be8-kube-api-access-lnvd6\") pod \"authorino-57f7f8659f-m9jd9\" (UID: \"effdddfb-3a3f-4eda-8115-21a0447f7be8\") " pod="kuadrant-system/authorino-57f7f8659f-m9jd9" Nov 02 09:39:58.468518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.468321 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" Nov 02 09:39:58.492881 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.492844 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:58.683823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.683795 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:58.756082 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.756001 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m9jd9"] Nov 02 09:39:58.776717 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:39:58.769099 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeffdddfb_3a3f_4eda_8115_21a0447f7be8.slice/crio-4dd32f8b89932370fb0e1870a965d5abe7c5f6f31a3c82fd91b0347bc44a8a6f WatchSource:0}: Error finding container 4dd32f8b89932370fb0e1870a965d5abe7c5f6f31a3c82fd91b0347bc44a8a6f: Status 404 returned error can't find the container with id 4dd32f8b89932370fb0e1870a965d5abe7c5f6f31a3c82fd91b0347bc44a8a6f Nov 02 09:39:58.873923 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:58.873866 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:39:59.492884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.492846 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" event={"ID":"effdddfb-3a3f-4eda-8115-21a0447f7be8","Type":"ContainerStarted","Data":"4dd32f8b89932370fb0e1870a965d5abe7c5f6f31a3c82fd91b0347bc44a8a6f"} Nov 02 09:39:59.498598 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.498566 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-ffv4m"] Nov 02 09:39:59.508243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.508216 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-ffv4m" Nov 02 09:39:59.512873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.512849 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wncj6\"" Nov 02 09:39:59.513441 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.513418 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-ffv4m"] Nov 02 09:39:59.563442 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.563411 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c5b0649-6d82-4523-bf72-118b0529d31a-bound-sa-token\") pod \"cert-manager-597b95cb84-ffv4m\" (UID: \"3c5b0649-6d82-4523-bf72-118b0529d31a\") " pod="cert-manager/cert-manager-597b95cb84-ffv4m" Nov 02 09:39:59.563643 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.563475 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b24dw\" (UniqueName: \"kubernetes.io/projected/3c5b0649-6d82-4523-bf72-118b0529d31a-kube-api-access-b24dw\") pod \"cert-manager-597b95cb84-ffv4m\" (UID: \"3c5b0649-6d82-4523-bf72-118b0529d31a\") " pod="cert-manager/cert-manager-597b95cb84-ffv4m" Nov 02 09:39:59.664685 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.664647 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b24dw\" (UniqueName: \"kubernetes.io/projected/3c5b0649-6d82-4523-bf72-118b0529d31a-kube-api-access-b24dw\") pod \"cert-manager-597b95cb84-ffv4m\" (UID: \"3c5b0649-6d82-4523-bf72-118b0529d31a\") " pod="cert-manager/cert-manager-597b95cb84-ffv4m" Nov 02 09:39:59.664858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.664740 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c5b0649-6d82-4523-bf72-118b0529d31a-bound-sa-token\") pod \"cert-manager-597b95cb84-ffv4m\" (UID: \"3c5b0649-6d82-4523-bf72-118b0529d31a\") " pod="cert-manager/cert-manager-597b95cb84-ffv4m" Nov 02 09:39:59.677983 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.677954 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c5b0649-6d82-4523-bf72-118b0529d31a-bound-sa-token\") pod \"cert-manager-597b95cb84-ffv4m\" (UID: \"3c5b0649-6d82-4523-bf72-118b0529d31a\") " pod="cert-manager/cert-manager-597b95cb84-ffv4m" Nov 02 09:39:59.678146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.678087 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b24dw\" (UniqueName: \"kubernetes.io/projected/3c5b0649-6d82-4523-bf72-118b0529d31a-kube-api-access-b24dw\") pod \"cert-manager-597b95cb84-ffv4m\" (UID: \"3c5b0649-6d82-4523-bf72-118b0529d31a\") " pod="cert-manager/cert-manager-597b95cb84-ffv4m" Nov 02 09:39:59.823094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:39:59.823054 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-ffv4m" Nov 02 09:40:00.103873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:00.103848 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-ffv4m"] Nov 02 09:40:00.106624 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:40:00.106601 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c5b0649_6d82_4523_bf72_118b0529d31a.slice/crio-25209311d6673a2bd9294f947796571d3453b7d4801092b0790441fcd5429ccf WatchSource:0}: Error finding container 25209311d6673a2bd9294f947796571d3453b7d4801092b0790441fcd5429ccf: Status 404 returned error can't find the container with id 25209311d6673a2bd9294f947796571d3453b7d4801092b0790441fcd5429ccf Nov 02 09:40:00.499583 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:00.499439 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" event={"ID":"cb0c1710-51cc-419a-84d3-f9e57666226c","Type":"ContainerStarted","Data":"d1ea600f0f4e14a9367ad518ef7d25ca7dcdb99cb3d094419706f5a0d93a86b8"} Nov 02 09:40:00.501753 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:00.501686 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-ffv4m" event={"ID":"3c5b0649-6d82-4523-bf72-118b0529d31a","Type":"ContainerStarted","Data":"f698b9c2f2af3dd18b2ab1896ecfa642d5360a2a6aa1af7c12148d399a126674"} Nov 02 09:40:00.501753 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:00.501726 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-ffv4m" event={"ID":"3c5b0649-6d82-4523-bf72-118b0529d31a","Type":"ContainerStarted","Data":"25209311d6673a2bd9294f947796571d3453b7d4801092b0790441fcd5429ccf"} Nov 02 09:40:00.518992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:00.518946 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c759d848c-x8mlt" podStartSLOduration=1.535792898 podStartE2EDuration="4.518930348s" totalImagesPullingTime="2.98313745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:00.517938443 +0000 UTC m=+279.357880438" watchObservedRunningTime="2026-11-02 09:40:00.518930348 +0000 UTC m=+279.358872339" Nov 02 09:40:00.536414 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:00.536366 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-ffv4m" podStartSLOduration=1.536351733 podStartE2EDuration="1.536351733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:00.535066493 +0000 UTC m=+279.375008489" watchObservedRunningTime="2026-11-02 09:40:00.536351733 +0000 UTC m=+279.376293726" Nov 02 09:40:02.512690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:02.512587 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" event={"ID":"effdddfb-3a3f-4eda-8115-21a0447f7be8","Type":"ContainerStarted","Data":"a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6"} Nov 02 09:40:02.538037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:02.537985 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" podStartSLOduration=1.246868263 podStartE2EDuration="4.53797038s" totalImagesPullingTime="3.291102117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:02.537045255 +0000 UTC m=+281.376987241" watchObservedRunningTime="2026-11-02 09:40:02.53797038 +0000 UTC m=+281.377912371" Nov 02 09:40:02.677656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:02.677628 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m9jd9"] Nov 02 09:40:03.489112 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:03.489081 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ztmqw" Nov 02 09:40:04.520705 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:04.520665 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" podUID="effdddfb-3a3f-4eda-8115-21a0447f7be8" containerName="authorino" containerID="cri-o://a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6" gracePeriod=30 Nov 02 09:40:04.771645 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:04.771579 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" Nov 02 09:40:04.912948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:04.912882 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effdddfb-3a3f-4eda-8115-21a0447f7be8-kube-api-access-lnvd6\" (UniqueName: \"kubernetes.io/projected/effdddfb-3a3f-4eda-8115-21a0447f7be8-kube-api-access-lnvd6\") pod \"effdddfb-3a3f-4eda-8115-21a0447f7be8\" (UID: \"effdddfb-3a3f-4eda-8115-21a0447f7be8\") " Nov 02 09:40:04.915239 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:04.915212 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effdddfb-3a3f-4eda-8115-21a0447f7be8-kube-api-access-lnvd6" pod "effdddfb-3a3f-4eda-8115-21a0447f7be8" (UID: "effdddfb-3a3f-4eda-8115-21a0447f7be8"). InnerVolumeSpecName "kube-api-access-lnvd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:40:05.014198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.014159 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnvd6\" (UniqueName: \"kubernetes.io/projected/effdddfb-3a3f-4eda-8115-21a0447f7be8-kube-api-access-lnvd6\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:40:05.299670 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.299637 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:05.478216 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.478184 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:05.526273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.526242 4467 generic.go:356] "Generic (PLEG): container finished" podID="effdddfb-3a3f-4eda-8115-21a0447f7be8" containerID="a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6" exitCode=0 Nov 02 09:40:05.526656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.526289 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" Nov 02 09:40:05.526656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.526310 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" event={"ID":"effdddfb-3a3f-4eda-8115-21a0447f7be8","Type":"ContainerDied","Data":"a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6"} Nov 02 09:40:05.526656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.526345 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m9jd9" event={"ID":"effdddfb-3a3f-4eda-8115-21a0447f7be8","Type":"ContainerDied","Data":"4dd32f8b89932370fb0e1870a965d5abe7c5f6f31a3c82fd91b0347bc44a8a6f"} Nov 02 09:40:05.526656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.526365 4467 scope.go:122] "RemoveContainer" containerID="a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6" Nov 02 09:40:05.534426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.534407 4467 scope.go:122] "RemoveContainer" containerID="a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6" Nov 02 09:40:05.534780 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:40:05.534757 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6\": container with ID starting with a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6 not found: ID does not exist" containerID="a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6" Nov 02 09:40:05.534846 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.534788 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6"} err="failed to get container status \"a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6\": rpc error: code = NotFound desc = could not find container \"a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6\": container with ID starting with a70081a7dd7d36c97b83e1f3a4a028686f5dfd9f9b17fbea5be0c3a4810fd5b6 not found: ID does not exist" Nov 02 09:40:05.544664 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.544634 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m9jd9"] Nov 02 09:40:05.549217 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.549189 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m9jd9"] Nov 02 09:40:05.660071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.660032 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:05.810825 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.810782 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh"] Nov 02 09:40:05.811180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.811166 4467 cpu_manager.go:414] "containerMap: removing container" podUID="effdddfb-3a3f-4eda-8115-21a0447f7be8" containerName="authorino" Nov 02 09:40:05.811235 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.811182 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effdddfb-3a3f-4eda-8115-21a0447f7be8" containerName="authorino" Nov 02 09:40:05.811269 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.811242 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="effdddfb-3a3f-4eda-8115-21a0447f7be8" containerName="authorino" Nov 02 09:40:05.815390 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.815372 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:05.829397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.829370 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh"] Nov 02 09:40:05.838626 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.838599 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 09:40:05.838772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.838687 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 09:40:05.839515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.839483 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9mq74\"" Nov 02 09:40:05.921421 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.921333 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94af169c-dda1-4cce-acdd-eb7070b2c30e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5njvh\" (UID: \"94af169c-dda1-4cce-acdd-eb7070b2c30e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:05.921421 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:05.921395 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txrww\" (UniqueName: \"kubernetes.io/projected/94af169c-dda1-4cce-acdd-eb7070b2c30e-kube-api-access-txrww\") pod \"kuadrant-operator-controller-manager-678d87766f-5njvh\" (UID: \"94af169c-dda1-4cce-acdd-eb7070b2c30e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:06.022301 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.022253 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txrww\" (UniqueName: \"kubernetes.io/projected/94af169c-dda1-4cce-acdd-eb7070b2c30e-kube-api-access-txrww\") pod \"kuadrant-operator-controller-manager-678d87766f-5njvh\" (UID: \"94af169c-dda1-4cce-acdd-eb7070b2c30e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:06.022449 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.022341 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94af169c-dda1-4cce-acdd-eb7070b2c30e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5njvh\" (UID: \"94af169c-dda1-4cce-acdd-eb7070b2c30e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:06.022723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.022706 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94af169c-dda1-4cce-acdd-eb7070b2c30e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5njvh\" (UID: \"94af169c-dda1-4cce-acdd-eb7070b2c30e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:06.038396 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.038368 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txrww\" (UniqueName: \"kubernetes.io/projected/94af169c-dda1-4cce-acdd-eb7070b2c30e-kube-api-access-txrww\") pod \"kuadrant-operator-controller-manager-678d87766f-5njvh\" (UID: \"94af169c-dda1-4cce-acdd-eb7070b2c30e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:06.127776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.127744 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:06.402415 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.402382 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh"] Nov 02 09:40:06.405558 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:40:06.405530 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94af169c_dda1_4cce_acdd_eb7070b2c30e.slice/crio-966f7e43c8e25c91b775e3ee2df7d7c5e03e798e9773593b95400026ac17b994 WatchSource:0}: Error finding container 966f7e43c8e25c91b775e3ee2df7d7c5e03e798e9773593b95400026ac17b994: Status 404 returned error can't find the container with id 966f7e43c8e25c91b775e3ee2df7d7c5e03e798e9773593b95400026ac17b994 Nov 02 09:40:06.495965 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.495833 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:06.531428 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.531395 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" event={"ID":"94af169c-dda1-4cce-acdd-eb7070b2c30e","Type":"ContainerStarted","Data":"966f7e43c8e25c91b775e3ee2df7d7c5e03e798e9773593b95400026ac17b994"} Nov 02 09:40:06.677807 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:06.677775 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:07.330239 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:07.330210 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effdddfb-3a3f-4eda-8115-21a0447f7be8" path="/var/lib/kubelet/pods/effdddfb-3a3f-4eda-8115-21a0447f7be8/volumes" Nov 02 09:40:10.232023 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:10.231994 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:11.984750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:11.984719 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:12.161485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:12.161448 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:12.561227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:12.561191 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" event={"ID":"94af169c-dda1-4cce-acdd-eb7070b2c30e","Type":"ContainerStarted","Data":"28b5dca9d63b33f59621b61c96e722e113c0284008ae724c83b06a0df3bb4162"} Nov 02 09:40:12.561471 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:12.561270 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:12.585535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:12.585487 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" podStartSLOduration=2.23827617 podStartE2EDuration="7.585472329s" totalImagesPullingTime="5.347196159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:12.585016355 +0000 UTC m=+291.424958351" watchObservedRunningTime="2026-11-02 09:40:12.585472329 +0000 UTC m=+291.425414314" Nov 02 09:40:14.552928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:14.552878 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:14.733353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:14.733321 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:17.861542 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:17.861496 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:18.084453 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.084420 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:40:18.087976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.087954 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:18.097165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.097140 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 09:40:18.097538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.097273 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zj9w7\"" Nov 02 09:40:18.102378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.102351 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:40:18.127054 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.126986 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:40:18.244451 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.244411 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2hd\" (UniqueName: \"kubernetes.io/projected/2506bfdc-b907-401c-ad87-d390372a5e08-kube-api-access-cm2hd\") pod \"limitador-limitador-7c9b87d574-52pxc\" (UID: \"2506bfdc-b907-401c-ad87-d390372a5e08\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:18.244451 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.244459 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2506bfdc-b907-401c-ad87-d390372a5e08-config-file\") pod \"limitador-limitador-7c9b87d574-52pxc\" (UID: \"2506bfdc-b907-401c-ad87-d390372a5e08\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:18.345644 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.345604 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2hd\" (UniqueName: \"kubernetes.io/projected/2506bfdc-b907-401c-ad87-d390372a5e08-kube-api-access-cm2hd\") pod \"limitador-limitador-7c9b87d574-52pxc\" (UID: \"2506bfdc-b907-401c-ad87-d390372a5e08\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:18.345823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.345654 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2506bfdc-b907-401c-ad87-d390372a5e08-config-file\") pod \"limitador-limitador-7c9b87d574-52pxc\" (UID: \"2506bfdc-b907-401c-ad87-d390372a5e08\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:18.346360 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.346340 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2506bfdc-b907-401c-ad87-d390372a5e08-config-file\") pod \"limitador-limitador-7c9b87d574-52pxc\" (UID: \"2506bfdc-b907-401c-ad87-d390372a5e08\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:18.358305 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.358275 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2hd\" (UniqueName: \"kubernetes.io/projected/2506bfdc-b907-401c-ad87-d390372a5e08-kube-api-access-cm2hd\") pod \"limitador-limitador-7c9b87d574-52pxc\" (UID: \"2506bfdc-b907-401c-ad87-d390372a5e08\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:18.404173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.404076 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:18.677332 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.677265 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:40:18.679492 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:40:18.679463 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2506bfdc_b907_401c_ad87_d390372a5e08.slice/crio-2b90b2a353a9b5e54c8b430177ce07b2303d680aa5e292d752ac889ae909069c WatchSource:0}: Error finding container 2b90b2a353a9b5e54c8b430177ce07b2303d680aa5e292d752ac889ae909069c: Status 404 returned error can't find the container with id 2b90b2a353a9b5e54c8b430177ce07b2303d680aa5e292d752ac889ae909069c Nov 02 09:40:18.716068 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:18.716028 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:19.179423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:19.179372 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:19.366189 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:19.366162 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:19.590327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:19.590291 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" event={"ID":"2506bfdc-b907-401c-ad87-d390372a5e08","Type":"ContainerStarted","Data":"2b90b2a353a9b5e54c8b430177ce07b2303d680aa5e292d752ac889ae909069c"} Nov 02 09:40:20.596525 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:20.596493 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" event={"ID":"2506bfdc-b907-401c-ad87-d390372a5e08","Type":"ContainerStarted","Data":"84530d1e1066cc336ea6f3bc21ae60a7a233c6cf8d6c5c2859b05bcfa8df040c"} Nov 02 09:40:20.596986 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:20.596648 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:20.619209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:20.619153 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" podStartSLOduration=1.499467018 podStartE2EDuration="2.619138862s" totalImagesPullingTime="1.119671844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:20.618299538 +0000 UTC m=+299.458241534" watchObservedRunningTime="2026-11-02 09:40:20.619138862 +0000 UTC m=+299.459080906" Nov 02 09:40:21.001918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:21.001771 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:21.183939 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:21.183871 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:23.567192 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:23.567154 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 09:40:24.003181 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:24.003090 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:24.232391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:24.232360 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:26.853519 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:40:26.853483 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:40:31.016509 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:31.016471 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:31.196669 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:31.196638 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:31.601858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:31.601823 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-52pxc" Nov 02 09:40:33.667438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:33.667355 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:33.844545 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:33.844512 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:39.024098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:39.024059 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:39.321735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:39.321700 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:39.505421 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:39.505388 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:43.674033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:43.673998 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:43.875355 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:43.875318 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:44.834870 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:44.834838 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:45.017228 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:45.017191 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:45.616386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:45.616352 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:50.748959 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:50.748719 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:50.937731 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:50.937689 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:54.256534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.256496 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2htfm"] Nov 02 09:40:54.266443 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.266417 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:54.271947 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.271298 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:54.272627 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.272598 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2htfm"] Nov 02 09:40:54.282366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.282340 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-d2kkp"] Nov 02 09:40:54.285914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.285879 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.295650 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.295623 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-d2kkp"] Nov 02 09:40:54.303925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.300079 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zs8xx\"" Nov 02 09:40:54.303925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.300374 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 09:40:54.303925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.300393 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 09:40:54.303925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.300437 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-8r7wc\"" Nov 02 09:40:54.303925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.300450 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 09:40:54.329046 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.329017 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 09:40:54.329046 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.329024 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 09:40:54.376538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.376493 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhtk\" (UniqueName: \"kubernetes.io/projected/77449e74-83c7-4592-bea4-48a7d20412d8-kube-api-access-jwhtk\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.376538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.376543 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77449e74-83c7-4592-bea4-48a7d20412d8-metrics-certs\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.376735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.376564 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadb4244-9782-409a-b4d2-e875d845c023-tls-certs\") pod \"model-serving-api-75bd89fdfc-2htfm\" (UID: \"eadb4244-9782-409a-b4d2-e875d845c023\") " pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:54.376773 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.376729 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dqg\" (UniqueName: \"kubernetes.io/projected/eadb4244-9782-409a-b4d2-e875d845c023-kube-api-access-q8dqg\") pod \"model-serving-api-75bd89fdfc-2htfm\" (UID: \"eadb4244-9782-409a-b4d2-e875d845c023\") " pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:54.376811 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.376776 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77449e74-83c7-4592-bea4-48a7d20412d8-cert\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.454263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.454216 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:54.477337 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.477303 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dqg\" (UniqueName: \"kubernetes.io/projected/eadb4244-9782-409a-b4d2-e875d845c023-kube-api-access-q8dqg\") pod \"model-serving-api-75bd89fdfc-2htfm\" (UID: \"eadb4244-9782-409a-b4d2-e875d845c023\") " pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:54.477515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.477350 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77449e74-83c7-4592-bea4-48a7d20412d8-cert\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.477515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.477392 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhtk\" (UniqueName: \"kubernetes.io/projected/77449e74-83c7-4592-bea4-48a7d20412d8-kube-api-access-jwhtk\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.477515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.477412 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77449e74-83c7-4592-bea4-48a7d20412d8-metrics-certs\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.477515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.477428 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadb4244-9782-409a-b4d2-e875d845c023-tls-certs\") pod \"model-serving-api-75bd89fdfc-2htfm\" (UID: \"eadb4244-9782-409a-b4d2-e875d845c023\") " pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:54.480042 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.480015 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77449e74-83c7-4592-bea4-48a7d20412d8-metrics-certs\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.480225 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.480203 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadb4244-9782-409a-b4d2-e875d845c023-tls-certs\") pod \"model-serving-api-75bd89fdfc-2htfm\" (UID: \"eadb4244-9782-409a-b4d2-e875d845c023\") " pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:54.480289 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.480207 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77449e74-83c7-4592-bea4-48a7d20412d8-cert\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.491342 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.491315 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhtk\" (UniqueName: \"kubernetes.io/projected/77449e74-83c7-4592-bea4-48a7d20412d8-kube-api-access-jwhtk\") pod \"odh-model-controller-769875db56-d2kkp\" (UID: \"77449e74-83c7-4592-bea4-48a7d20412d8\") " pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.491481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.491388 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dqg\" (UniqueName: \"kubernetes.io/projected/eadb4244-9782-409a-b4d2-e875d845c023-kube-api-access-q8dqg\") pod \"model-serving-api-75bd89fdfc-2htfm\" (UID: \"eadb4244-9782-409a-b4d2-e875d845c023\") " pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:54.583701 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.583665 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:54.601709 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.601678 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:54.661273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.661031 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:54.864576 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.864509 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2htfm"] Nov 02 09:40:54.867103 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:40:54.867080 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadb4244_9782_409a_b4d2_e875d845c023.slice/crio-eb963634f3827f105c7ab8ffe0e47d0236e18ce93ae3aa7bd8319e58741342e4 WatchSource:0}: Error finding container eb963634f3827f105c7ab8ffe0e47d0236e18ce93ae3aa7bd8319e58741342e4: Status 404 returned error can't find the container with id eb963634f3827f105c7ab8ffe0e47d0236e18ce93ae3aa7bd8319e58741342e4 Nov 02 09:40:54.869068 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.869047 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 09:40:54.892884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:54.892855 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-d2kkp"] Nov 02 09:40:54.895960 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:40:54.895934 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77449e74_83c7_4592_bea4_48a7d20412d8.slice/crio-cf99e33ad3fd70bd38e8b4e4b0d8d4b57b7c3fcc2a2e4497e9d1872afcf191a8 WatchSource:0}: Error finding container cf99e33ad3fd70bd38e8b4e4b0d8d4b57b7c3fcc2a2e4497e9d1872afcf191a8: Status 404 returned error can't find the container with id cf99e33ad3fd70bd38e8b4e4b0d8d4b57b7c3fcc2a2e4497e9d1872afcf191a8 Nov 02 09:40:55.449184 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:55.449147 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:55.630912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:55.630800 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:55.771104 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:55.770969 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-d2kkp" event={"ID":"77449e74-83c7-4592-bea4-48a7d20412d8","Type":"ContainerStarted","Data":"cf99e33ad3fd70bd38e8b4e4b0d8d4b57b7c3fcc2a2e4497e9d1872afcf191a8"} Nov 02 09:40:55.774404 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:55.774286 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" event={"ID":"eadb4244-9782-409a-b4d2-e875d845c023","Type":"ContainerStarted","Data":"eb963634f3827f105c7ab8ffe0e47d0236e18ce93ae3aa7bd8319e58741342e4"} Nov 02 09:40:55.837931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:55.828169 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:56.869041 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:40:56.869003 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:40:56.988758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:56.988716 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:40:57.786971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:57.786927 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" event={"ID":"eadb4244-9782-409a-b4d2-e875d845c023","Type":"ContainerStarted","Data":"76e94ebda6e55879d8b901e4673d79bd020ecd1fc12a463716f812670291806d"} Nov 02 09:40:57.787161 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:57.787078 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:40:57.808717 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:57.807598 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" podStartSLOduration=1.437464374 podStartE2EDuration="3.807581802s" totalImagesPullingTime="2.370117428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:57.805989799 +0000 UTC m=+336.645931795" watchObservedRunningTime="2026-11-02 09:40:57.807581802 +0000 UTC m=+336.647523800" Nov 02 09:40:58.792485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:58.792441 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-d2kkp" event={"ID":"77449e74-83c7-4592-bea4-48a7d20412d8","Type":"ContainerStarted","Data":"fccfc194ac38caa956428d532e7a66b4e0002f4fa740daece489e21e8428066d"} Nov 02 09:40:58.792988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:58.792802 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:40:58.813755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:40:58.813700 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-d2kkp" podStartSLOduration=1.822391531 podStartE2EDuration="4.813687447s" totalImagesPullingTime="2.991295916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:58.811845397 +0000 UTC m=+337.651787392" watchObservedRunningTime="2026-11-02 09:40:58.813687447 +0000 UTC m=+337.653629442" Nov 02 09:41:00.171383 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:00.171345 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:00.408320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:00.408254 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:07.429696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:07.429657 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:07.614070 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:07.614037 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:07.792347 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:07.792262 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:08.796176 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:08.796142 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-2htfm" Nov 02 09:41:09.798497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:09.798462 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-d2kkp" Nov 02 09:41:10.282751 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:10.282718 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:12.209314 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:12.209281 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:12.395228 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:12.395190 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:13.662120 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.662081 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp"] Nov 02 09:41:13.665849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.665825 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.671476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.671447 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 09:41:13.672402 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.672380 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 09:41:13.672537 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.672421 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qsrdf\"" Nov 02 09:41:13.672537 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.672422 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 09:41:13.680052 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.679961 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp"] Nov 02 09:41:13.752766 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.752726 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/925f1379-43e8-4594-ae67-ae2cf3bafab3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.752766 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.752765 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/925f1379-43e8-4594-ae67-ae2cf3bafab3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.753031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.752859 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/925f1379-43e8-4594-ae67-ae2cf3bafab3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.753031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.752930 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmczw\" (UniqueName: \"kubernetes.io/projected/925f1379-43e8-4594-ae67-ae2cf3bafab3-kube-api-access-gmczw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.810557 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.810523 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:13.854101 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.854059 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmczw\" (UniqueName: \"kubernetes.io/projected/925f1379-43e8-4594-ae67-ae2cf3bafab3-kube-api-access-gmczw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.854282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.854142 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/925f1379-43e8-4594-ae67-ae2cf3bafab3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.854282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.854169 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/925f1379-43e8-4594-ae67-ae2cf3bafab3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.854282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.854206 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/925f1379-43e8-4594-ae67-ae2cf3bafab3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.854921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.854873 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/925f1379-43e8-4594-ae67-ae2cf3bafab3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.855206 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.855186 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/925f1379-43e8-4594-ae67-ae2cf3bafab3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.856688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.856664 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/925f1379-43e8-4594-ae67-ae2cf3bafab3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.880234 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.880210 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmczw\" (UniqueName: \"kubernetes.io/projected/925f1379-43e8-4594-ae67-ae2cf3bafab3-kube-api-access-gmczw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7shdp\" (UID: \"925f1379-43e8-4594-ae67-ae2cf3bafab3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:13.982284 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:13.982240 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:14.010330 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:14.010293 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:14.261965 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:14.261873 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp"] Nov 02 09:41:14.265951 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:41:14.265917 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925f1379_43e8_4594_ae67_ae2cf3bafab3.slice/crio-ad0f585531fe70f2565f20dcbd86fbeab7016f15625ec72afb85f16fcac89a73 WatchSource:0}: Error finding container ad0f585531fe70f2565f20dcbd86fbeab7016f15625ec72afb85f16fcac89a73: Status 404 returned error can't find the container with id ad0f585531fe70f2565f20dcbd86fbeab7016f15625ec72afb85f16fcac89a73 Nov 02 09:41:14.862634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:14.862594 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" event={"ID":"925f1379-43e8-4594-ae67-ae2cf3bafab3","Type":"ContainerStarted","Data":"ad0f585531fe70f2565f20dcbd86fbeab7016f15625ec72afb85f16fcac89a73"} Nov 02 09:41:26.856422 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:41:26.856381 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:41:26.924384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:26.924354 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" event={"ID":"925f1379-43e8-4594-ae67-ae2cf3bafab3","Type":"ContainerStarted","Data":"b1de5b84862b3b0c3b9234d056f3e789b9eb6ca167ecd1bee746bedf3781ab74"} Nov 02 09:41:26.924545 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:26.924454 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:26.945392 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:26.945347 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" podStartSLOduration=1.390215137 podStartE2EDuration="13.945335395s" totalImagesPullingTime="12.555120258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:41:26.944203563 +0000 UTC m=+365.784145559" watchObservedRunningTime="2026-11-02 09:41:26.945335395 +0000 UTC m=+365.785277393" Nov 02 09:41:37.933938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:37.933907 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7shdp" Nov 02 09:41:44.347141 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:44.347103 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:44.533272 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:44.533236 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:45.636456 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:45.636414 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:45.824851 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:45.824810 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:46.528542 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:46.528510 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:41:47.583498 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:47.583461 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:47.768731 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:47.768658 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:56.641204 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:56.641166 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:56.855583 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:41:56.855551 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:41:56.920154 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:56.920049 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:41:57.101175 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.101144 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj"] Nov 02 09:41:57.106937 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.106912 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.114190 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.114129 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 09:41:57.114327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.114271 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 09:41:57.115242 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.115219 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5thw\"" Nov 02 09:41:57.115480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.115439 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 09:41:57.115480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.115468 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 09:41:57.116382 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.116354 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj"] Nov 02 09:41:57.230173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.230082 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h77m\" (UniqueName: \"kubernetes.io/projected/5e201e45-6561-430f-b58c-325fb4c615d8-kube-api-access-6h77m\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.230173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.230127 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e201e45-6561-430f-b58c-325fb4c615d8-openshift-service-ca\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.230173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.230145 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-metrics-certs\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.230427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.230285 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-maas-api-tls\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.331320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.331282 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h77m\" (UniqueName: \"kubernetes.io/projected/5e201e45-6561-430f-b58c-325fb4c615d8-kube-api-access-6h77m\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.331515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.331333 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e201e45-6561-430f-b58c-325fb4c615d8-openshift-service-ca\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.331515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.331360 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-metrics-certs\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.331515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.331391 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-maas-api-tls\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.332014 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.331987 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e201e45-6561-430f-b58c-325fb4c615d8-openshift-service-ca\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.333968 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.333946 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-maas-api-tls\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.334056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.333947 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-metrics-certs\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.348847 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.348820 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h77m\" (UniqueName: \"kubernetes.io/projected/5e201e45-6561-430f-b58c-325fb4c615d8-kube-api-access-6h77m\") pod \"maas-api-7ff9894dcf-5rkrj\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.420547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.420505 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:41:57.693656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:57.693585 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj"] Nov 02 09:41:57.696373 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:41:57.696336 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e201e45_6561_430f_b58c_325fb4c615d8.slice/crio-3089771f5e6f055424e7231e505c909a78890e5738001ecad333787a492e829d WatchSource:0}: Error finding container 3089771f5e6f055424e7231e505c909a78890e5738001ecad333787a492e829d: Status 404 returned error can't find the container with id 3089771f5e6f055424e7231e505c909a78890e5738001ecad333787a492e829d Nov 02 09:41:58.063336 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:41:58.063303 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" event={"ID":"5e201e45-6561-430f-b58c-325fb4c615d8","Type":"ContainerStarted","Data":"3089771f5e6f055424e7231e505c909a78890e5738001ecad333787a492e829d"} Nov 02 09:42:01.086060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:01.086022 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" event={"ID":"5e201e45-6561-430f-b58c-325fb4c615d8","Type":"ContainerStarted","Data":"9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94"} Nov 02 09:42:01.086501 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:01.086167 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:42:01.108676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:01.108624 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" podStartSLOduration=1.532340472 podStartE2EDuration="4.108608349s" totalImagesPullingTime="2.576267877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:42:01.108330629 +0000 UTC m=+399.948272625" watchObservedRunningTime="2026-11-02 09:42:01.108608349 +0000 UTC m=+399.948550345" Nov 02 09:42:07.095861 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:07.095783 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 09:42:10.344399 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:10.344350 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:42:26.853435 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:42:26.853398 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:42:28.009317 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:28.009285 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:42:34.419754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.419714 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pb8cr"] Nov 02 09:42:34.423101 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.423079 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:34.432085 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.432058 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-nccqc\"" Nov 02 09:42:34.432311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.432286 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 09:42:34.432419 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.432332 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 09:42:34.433245 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.433212 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pb8cr"] Nov 02 09:42:34.457076 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.457044 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-pb8cr\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:34.457297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.457098 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7lvf\" (UniqueName: \"kubernetes.io/projected/effee251-37f5-422a-a827-99c497f75f9f-kube-api-access-b7lvf\") pod \"ai-gateway-controller-7d85675f98-pb8cr\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:34.496871 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.496837 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:42:34.557802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.557768 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-pb8cr\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:34.558015 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.557811 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7lvf\" (UniqueName: \"kubernetes.io/projected/effee251-37f5-422a-a827-99c497f75f9f-kube-api-access-b7lvf\") pod \"ai-gateway-controller-7d85675f98-pb8cr\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:34.558015 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:42:34.557941 4467 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found Nov 02 09:42:34.558015 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:42:34.558009 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs podName:effee251-37f5-422a-a827-99c497f75f9f nodeName:}" failed. No retries permitted until 2026-11-02 09:42:35.057993402 +0000 UTC m=+433.897935378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs") pod "ai-gateway-controller-7d85675f98-pb8cr" (UID: "effee251-37f5-422a-a827-99c497f75f9f") : secret "ai-gateway-controller-metrics-tls" not found Nov 02 09:42:34.571022 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:34.570989 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7lvf\" (UniqueName: \"kubernetes.io/projected/effee251-37f5-422a-a827-99c497f75f9f-kube-api-access-b7lvf\") pod \"ai-gateway-controller-7d85675f98-pb8cr\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:35.006976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:35.006941 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:42:35.062390 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:35.062359 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-pb8cr\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:35.064741 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:35.064723 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-pb8cr\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:35.337213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:35.337186 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:35.608976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:35.608948 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pb8cr"] Nov 02 09:42:35.611557 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:42:35.611533 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffee251_37f5_422a_a827_99c497f75f9f.slice/crio-5e8ea8fb51d07c82a5b227d4d0b3f9feea61b1b8783577bf6c017018cb08dede WatchSource:0}: Error finding container 5e8ea8fb51d07c82a5b227d4d0b3f9feea61b1b8783577bf6c017018cb08dede: Status 404 returned error can't find the container with id 5e8ea8fb51d07c82a5b227d4d0b3f9feea61b1b8783577bf6c017018cb08dede Nov 02 09:42:36.254036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:36.253999 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" event={"ID":"effee251-37f5-422a-a827-99c497f75f9f","Type":"ContainerStarted","Data":"5e8ea8fb51d07c82a5b227d4d0b3f9feea61b1b8783577bf6c017018cb08dede"} Nov 02 09:42:39.195381 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.195346 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sbd6p"] Nov 02 09:42:39.198833 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.198811 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:39.208203 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.208176 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 09:42:39.209238 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.209217 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ftlll\"" Nov 02 09:42:39.210761 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.210735 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sbd6p"] Nov 02 09:42:39.256021 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.255986 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:42:39.269248 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.269217 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" event={"ID":"effee251-37f5-422a-a827-99c497f75f9f","Type":"ContainerStarted","Data":"8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268"} Nov 02 09:42:39.269396 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.269336 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:39.293672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.293623 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" podStartSLOduration=2.432070785 podStartE2EDuration="5.293609443s" totalImagesPullingTime="2.861538658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:42:39.29208925 +0000 UTC m=+438.132031246" watchObservedRunningTime="2026-11-02 09:42:39.293609443 +0000 UTC m=+438.133551437" Nov 02 09:42:39.302384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.302358 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-sbd6p\" (UID: \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:39.302522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.302502 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlt5\" (UniqueName: \"kubernetes.io/projected/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-kube-api-access-lnlt5\") pod \"payload-processing-55fbbbf9dc-sbd6p\" (UID: \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:39.403650 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.403614 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlt5\" (UniqueName: \"kubernetes.io/projected/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-kube-api-access-lnlt5\") pod \"payload-processing-55fbbbf9dc-sbd6p\" (UID: \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:39.403846 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.403663 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-sbd6p\" (UID: \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:39.404407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.404381 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-sbd6p\" (UID: \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:39.417311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.417283 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlt5\" (UniqueName: \"kubernetes.io/projected/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-kube-api-access-lnlt5\") pod \"payload-processing-55fbbbf9dc-sbd6p\" (UID: \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:39.444266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.444238 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:42:39.512270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.512204 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:39.784973 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:39.784917 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sbd6p"] Nov 02 09:42:39.787703 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:42:39.787676 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe0b6eb_e661_4ac4_ac42_ac13e163f5de.slice/crio-81125318615b20dfb6c1e69718bb0ee287299503bbaa012268a5d68135c25ded WatchSource:0}: Error finding container 81125318615b20dfb6c1e69718bb0ee287299503bbaa012268a5d68135c25ded: Status 404 returned error can't find the container with id 81125318615b20dfb6c1e69718bb0ee287299503bbaa012268a5d68135c25ded Nov 02 09:42:40.274669 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:40.274631 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" event={"ID":"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de","Type":"ContainerStarted","Data":"81125318615b20dfb6c1e69718bb0ee287299503bbaa012268a5d68135c25ded"} Nov 02 09:42:43.290504 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:43.290469 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" event={"ID":"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de","Type":"ContainerStarted","Data":"db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8"} Nov 02 09:42:43.290958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:43.290566 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:43.310420 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:43.310370 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" podStartSLOduration=1.790652311 podStartE2EDuration="4.310357396s" totalImagesPullingTime="2.519705085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:42:43.308149917 +0000 UTC m=+442.148091922" watchObservedRunningTime="2026-11-02 09:42:43.310357396 +0000 UTC m=+442.150299391" Nov 02 09:42:49.992242 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:49.992200 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:42:50.276540 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.276450 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:42:50.556488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.556414 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6f755f6879-xh4h5"] Nov 02 09:42:50.560055 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.560034 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:50.575204 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.575177 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6f755f6879-xh4h5"] Nov 02 09:42:50.700433 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.700396 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwhl\" (UniqueName: \"kubernetes.io/projected/30722de2-3a23-49f9-9672-50446962e7e2-kube-api-access-xgwhl\") pod \"ai-gateway-controller-6f755f6879-xh4h5\" (UID: \"30722de2-3a23-49f9-9672-50446962e7e2\") " pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:50.700599 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.700458 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30722de2-3a23-49f9-9672-50446962e7e2-metrics-certs\") pod \"ai-gateway-controller-6f755f6879-xh4h5\" (UID: \"30722de2-3a23-49f9-9672-50446962e7e2\") " pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:50.800878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.800840 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30722de2-3a23-49f9-9672-50446962e7e2-metrics-certs\") pod \"ai-gateway-controller-6f755f6879-xh4h5\" (UID: \"30722de2-3a23-49f9-9672-50446962e7e2\") " pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:50.801052 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.800928 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwhl\" (UniqueName: \"kubernetes.io/projected/30722de2-3a23-49f9-9672-50446962e7e2-kube-api-access-xgwhl\") pod \"ai-gateway-controller-6f755f6879-xh4h5\" (UID: \"30722de2-3a23-49f9-9672-50446962e7e2\") " pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:50.803396 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.803372 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30722de2-3a23-49f9-9672-50446962e7e2-metrics-certs\") pod \"ai-gateway-controller-6f755f6879-xh4h5\" (UID: \"30722de2-3a23-49f9-9672-50446962e7e2\") " pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:50.813424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.813400 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwhl\" (UniqueName: \"kubernetes.io/projected/30722de2-3a23-49f9-9672-50446962e7e2-kube-api-access-xgwhl\") pod \"ai-gateway-controller-6f755f6879-xh4h5\" (UID: \"30722de2-3a23-49f9-9672-50446962e7e2\") " pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:50.873012 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:50.872981 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:51.149916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:51.149770 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6f755f6879-xh4h5"] Nov 02 09:42:51.152663 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:42:51.152640 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30722de2_3a23_49f9_9672_50446962e7e2.slice/crio-703d7bd3fdf22bb5c26be378abc78b11374108a802db56cb62bd68acf95a4936 WatchSource:0}: Error finding container 703d7bd3fdf22bb5c26be378abc78b11374108a802db56cb62bd68acf95a4936: Status 404 returned error can't find the container with id 703d7bd3fdf22bb5c26be378abc78b11374108a802db56cb62bd68acf95a4936 Nov 02 09:42:51.330107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:51.330074 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" event={"ID":"30722de2-3a23-49f9-9672-50446962e7e2","Type":"ContainerStarted","Data":"703d7bd3fdf22bb5c26be378abc78b11374108a802db56cb62bd68acf95a4936"} Nov 02 09:42:52.333077 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:52.333038 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" event={"ID":"30722de2-3a23-49f9-9672-50446962e7e2","Type":"ContainerStarted","Data":"2dbe8d0688ebdda8527b6d33fc17072652c73100065fad7a294a8eada5e5fcec"} Nov 02 09:42:52.333539 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:52.333124 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:42:52.353042 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:52.352994 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" podStartSLOduration=1.930803654 podStartE2EDuration="2.352980829s" totalImagesPullingTime="422.177175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:42:52.352203023 +0000 UTC m=+451.192145023" watchObservedRunningTime="2026-11-02 09:42:52.352980829 +0000 UTC m=+451.192922823" Nov 02 09:42:54.298321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:42:54.298295 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 09:42:56.854330 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:42:56.854280 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:43:03.345691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.345661 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6f755f6879-xh4h5" Nov 02 09:43:03.396189 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.396153 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pb8cr"] Nov 02 09:43:03.396433 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.396407 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" podUID="effee251-37f5-422a-a827-99c497f75f9f" containerName="manager" containerID="cri-o://8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268" gracePeriod=30 Nov 02 09:43:03.633726 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.633702 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:43:03.720814 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.720780 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effee251-37f5-422a-a827-99c497f75f9f-kube-api-access-b7lvf\" (UniqueName: \"kubernetes.io/projected/effee251-37f5-422a-a827-99c497f75f9f-kube-api-access-b7lvf\") pod \"effee251-37f5-422a-a827-99c497f75f9f\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " Nov 02 09:43:03.721005 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.720865 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs\") pod \"effee251-37f5-422a-a827-99c497f75f9f\" (UID: \"effee251-37f5-422a-a827-99c497f75f9f\") " Nov 02 09:43:03.723136 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.723110 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effee251-37f5-422a-a827-99c497f75f9f-kube-api-access-b7lvf" pod "effee251-37f5-422a-a827-99c497f75f9f" (UID: "effee251-37f5-422a-a827-99c497f75f9f"). InnerVolumeSpecName "kube-api-access-b7lvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:43:03.723221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.723199 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs" pod "effee251-37f5-422a-a827-99c497f75f9f" (UID: "effee251-37f5-422a-a827-99c497f75f9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:43:03.822118 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.822077 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effee251-37f5-422a-a827-99c497f75f9f-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:43:03.822118 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:03.822111 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7lvf\" (UniqueName: \"kubernetes.io/projected/effee251-37f5-422a-a827-99c497f75f9f-kube-api-access-b7lvf\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:43:04.393015 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.392979 4467 generic.go:356] "Generic (PLEG): container finished" podID="effee251-37f5-422a-a827-99c497f75f9f" containerID="8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268" exitCode=0 Nov 02 09:43:04.393478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.393030 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" Nov 02 09:43:04.393478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.393064 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" event={"ID":"effee251-37f5-422a-a827-99c497f75f9f","Type":"ContainerDied","Data":"8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268"} Nov 02 09:43:04.393478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.393111 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-pb8cr" event={"ID":"effee251-37f5-422a-a827-99c497f75f9f","Type":"ContainerDied","Data":"5e8ea8fb51d07c82a5b227d4d0b3f9feea61b1b8783577bf6c017018cb08dede"} Nov 02 09:43:04.393478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.393133 4467 scope.go:122] "RemoveContainer" containerID="8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268" Nov 02 09:43:04.401840 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.401651 4467 scope.go:122] "RemoveContainer" containerID="8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268" Nov 02 09:43:04.401986 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:43:04.401966 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268\": container with ID starting with 8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268 not found: ID does not exist" containerID="8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268" Nov 02 09:43:04.402033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.401994 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268"} err="failed to get container status \"8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268\": rpc error: code = NotFound desc = could not find container \"8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268\": container with ID starting with 8fd964b170ce473020be27cf595b5cc3d3edd90629cbf22228d280d035bcf268 not found: ID does not exist" Nov 02 09:43:04.415533 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.415511 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pb8cr"] Nov 02 09:43:04.419836 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:04.419816 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-pb8cr"] Nov 02 09:43:05.315431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:05.315398 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:05.329680 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:05.329652 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effee251-37f5-422a-a827-99c497f75f9f" path="/var/lib/kubelet/pods/effee251-37f5-422a-a827-99c497f75f9f/volumes" Nov 02 09:43:05.959269 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:05.959236 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:07.184517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:07.184484 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:07.394071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:07.394038 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:10.402488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.402448 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:10.582963 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.582934 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j"] Nov 02 09:43:10.583329 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.583315 4467 cpu_manager.go:414] "containerMap: removing container" podUID="effee251-37f5-422a-a827-99c497f75f9f" containerName="manager" Nov 02 09:43:10.583382 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.583331 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effee251-37f5-422a-a827-99c497f75f9f" containerName="manager" Nov 02 09:43:10.583419 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.583391 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="effee251-37f5-422a-a827-99c497f75f9f" containerName="manager" Nov 02 09:43:10.588016 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.587996 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.596631 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.596609 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 09:43:10.596747 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.596735 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 09:43:10.597533 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.597511 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cbg4l\"" Nov 02 09:43:10.597647 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.597585 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 09:43:10.599541 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.599521 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j"] Nov 02 09:43:10.679996 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.679884 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.679996 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.679947 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.680179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.680036 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.680179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.680070 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4fd49f6-ec97-487c-aebe-369ad16468d7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.680179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.680089 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.680179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.680105 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s456n\" (UniqueName: \"kubernetes.io/projected/b4fd49f6-ec97-487c-aebe-369ad16468d7-kube-api-access-s456n\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.734597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.734563 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:10.781120 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781089 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.781263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781127 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.781263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781175 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.781263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781199 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4fd49f6-ec97-487c-aebe-369ad16468d7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.781263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781218 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.781263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781245 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s456n\" (UniqueName: \"kubernetes.io/projected/b4fd49f6-ec97-487c-aebe-369ad16468d7-kube-api-access-s456n\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.781576 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781547 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.781733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781600 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.781733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.781678 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.783398 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.783377 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4fd49f6-ec97-487c-aebe-369ad16468d7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.783746 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.783729 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4fd49f6-ec97-487c-aebe-369ad16468d7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.794390 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.794363 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s456n\" (UniqueName: \"kubernetes.io/projected/b4fd49f6-ec97-487c-aebe-369ad16468d7-kube-api-access-s456n\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h557j\" (UID: \"b4fd49f6-ec97-487c-aebe-369ad16468d7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:10.901495 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:10.901463 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:11.283594 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:11.283520 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j"] Nov 02 09:43:11.286596 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:43:11.286569 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fd49f6_ec97_487c_aebe_369ad16468d7.slice/crio-c7a9fe0bb0d7dffc4e4cb0b9af244e6b18d1d0d2a9f718943fdcd1442173565f WatchSource:0}: Error finding container c7a9fe0bb0d7dffc4e4cb0b9af244e6b18d1d0d2a9f718943fdcd1442173565f: Status 404 returned error can't find the container with id c7a9fe0bb0d7dffc4e4cb0b9af244e6b18d1d0d2a9f718943fdcd1442173565f Nov 02 09:43:11.320609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:11.320539 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:43:11.424448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:11.424416 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" event={"ID":"b4fd49f6-ec97-487c-aebe-369ad16468d7","Type":"ContainerStarted","Data":"c7a9fe0bb0d7dffc4e4cb0b9af244e6b18d1d0d2a9f718943fdcd1442173565f"} Nov 02 09:43:12.769027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:12.769004 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:13.435649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:13.435612 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" event={"ID":"b4fd49f6-ec97-487c-aebe-369ad16468d7","Type":"ContainerStarted","Data":"266249b4c5c600bf1763ca00c52650b220b6eb72f7fb35537e366c0d1d17c555"} Nov 02 09:43:13.458193 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:13.458143 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" podStartSLOduration=1.99170144 podStartE2EDuration="3.45812907s" totalImagesPullingTime="1.46642763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:13.456657131 +0000 UTC m=+472.296599129" watchObservedRunningTime="2026-11-02 09:43:13.45812907 +0000 UTC m=+472.298071068" Nov 02 09:43:13.902374 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:13.902339 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:13.930427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:13.930395 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:14.426234 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:14.426197 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:43:15.420755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.420721 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk"] Nov 02 09:43:15.424401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.424377 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.429300 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.429279 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 09:43:15.436697 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.436673 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk"] Nov 02 09:43:15.520983 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.520950 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3162dd91-0c87-4098-9c76-16fe6ef4b17b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.521166 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.521065 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.521166 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.521147 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdbn\" (UniqueName: \"kubernetes.io/projected/3162dd91-0c87-4098-9c76-16fe6ef4b17b-kube-api-access-6vdbn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.521254 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.521233 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.521327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.521306 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.521378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.521356 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.613838 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.613804 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:15.622629 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.622599 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.622742 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.622639 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.622742 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.622663 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.622839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.622742 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3162dd91-0c87-4098-9c76-16fe6ef4b17b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.622839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.622803 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.622988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.622836 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdbn\" (UniqueName: \"kubernetes.io/projected/3162dd91-0c87-4098-9c76-16fe6ef4b17b-kube-api-access-6vdbn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.623057 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.623026 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.623174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.623117 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.623375 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.623353 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.625326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.625305 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3162dd91-0c87-4098-9c76-16fe6ef4b17b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.625414 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.625313 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3162dd91-0c87-4098-9c76-16fe6ef4b17b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.635302 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.635280 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdbn\" (UniqueName: \"kubernetes.io/projected/3162dd91-0c87-4098-9c76-16fe6ef4b17b-kube-api-access-6vdbn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xg2gk\" (UID: \"3162dd91-0c87-4098-9c76-16fe6ef4b17b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:15.738514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:15.738440 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:16.014101 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:16.014074 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk"] Nov 02 09:43:16.016792 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:43:16.016766 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3162dd91_0c87_4098_9c76_16fe6ef4b17b.slice/crio-11dbdf90c7a49d6241a1b76bbc3e3ea738a8049de0b463bc851fd0fe0b767599 WatchSource:0}: Error finding container 11dbdf90c7a49d6241a1b76bbc3e3ea738a8049de0b463bc851fd0fe0b767599: Status 404 returned error can't find the container with id 11dbdf90c7a49d6241a1b76bbc3e3ea738a8049de0b463bc851fd0fe0b767599 Nov 02 09:43:16.023775 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:16.023719 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:16.438554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:16.438519 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:43:16.452381 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:16.452345 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" event={"ID":"3162dd91-0c87-4098-9c76-16fe6ef4b17b","Type":"ContainerStarted","Data":"3b8dd1673d102ac89b0951c8717acfea9bed55371c67f37a440c4a831cb56b14"} Nov 02 09:43:16.452539 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:16.452398 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" event={"ID":"3162dd91-0c87-4098-9c76-16fe6ef4b17b","Type":"ContainerStarted","Data":"11dbdf90c7a49d6241a1b76bbc3e3ea738a8049de0b463bc851fd0fe0b767599"} Nov 02 09:43:16.479505 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:16.479451 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" podStartSLOduration=1.189441371 podStartE2EDuration="1.479437744s" totalImagesPullingTime="289.996373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:16.478462823 +0000 UTC m=+475.318404817" watchObservedRunningTime="2026-11-02 09:43:16.479437744 +0000 UTC m=+475.319379739" Nov 02 09:43:16.739548 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:16.739469 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:17.528358 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.528320 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf"] Nov 02 09:43:17.532227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.532208 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.541089 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.540732 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 09:43:17.546262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.546233 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf"] Nov 02 09:43:17.639907 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.639858 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.639907 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.639909 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82k9\" (UniqueName: \"kubernetes.io/projected/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-kube-api-access-g82k9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.640124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.640005 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.640124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.640051 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.640124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.640079 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.640234 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.640173 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.666666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.666607 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:17.741555 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.741521 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.741555 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.741555 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82k9\" (UniqueName: \"kubernetes.io/projected/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-kube-api-access-g82k9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.741804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.741579 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.741804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.741595 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.741804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.741732 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.741986 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.741853 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.742036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.742013 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.742094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.742076 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.742147 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.742134 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.743967 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.743948 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.744272 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.744254 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.753913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.753878 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82k9\" (UniqueName: \"kubernetes.io/projected/91fbf792-c0c1-4431-8ea0-e288a4ec58c4-kube-api-access-g82k9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf\" (UID: \"91fbf792-c0c1-4431-8ea0-e288a4ec58c4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:17.849440 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:17.849409 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:18.137225 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:18.137127 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf"] Nov 02 09:43:18.139924 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:43:18.139885 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fbf792_c0c1_4431_8ea0_e288a4ec58c4.slice/crio-81206153c161919ba92bbd85eaa1b75dfc97c8ac9611ffade75e1707887edf5f WatchSource:0}: Error finding container 81206153c161919ba92bbd85eaa1b75dfc97c8ac9611ffade75e1707887edf5f: Status 404 returned error can't find the container with id 81206153c161919ba92bbd85eaa1b75dfc97c8ac9611ffade75e1707887edf5f Nov 02 09:43:18.464099 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:18.464069 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" event={"ID":"91fbf792-c0c1-4431-8ea0-e288a4ec58c4","Type":"ContainerStarted","Data":"81206153c161919ba92bbd85eaa1b75dfc97c8ac9611ffade75e1707887edf5f"} Nov 02 09:43:19.256755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:19.256720 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:19.470026 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:19.469994 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" event={"ID":"91fbf792-c0c1-4431-8ea0-e288a4ec58c4","Type":"ContainerStarted","Data":"96ec9c51e9cd8659e345720e3784314b81c04c13f00b41bcc435195d9026f286"} Nov 02 09:43:19.492539 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:19.492480 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" podStartSLOduration=2.215510336 podStartE2EDuration="2.492466336s" totalImagesPullingTime="276.956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:19.491331845 +0000 UTC m=+478.331273842" watchObservedRunningTime="2026-11-02 09:43:19.492466336 +0000 UTC m=+478.332408330" Nov 02 09:43:19.849935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:19.849899 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:20.128942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:20.128842 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:43:20.230257 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:20.230225 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:20.902690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:20.902649 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:20.915651 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:20.915627 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:21.491978 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:21.491944 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h557j" Nov 02 09:43:22.661031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:22.660997 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:24.157777 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:24.157731 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:43:25.739075 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:25.739037 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:25.752093 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:25.752060 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:26.516842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:26.516812 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xg2gk" Nov 02 09:43:26.854687 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:43:26.854651 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:43:27.850121 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:27.850087 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:27.862738 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:27.862716 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:28.526370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:28.526343 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf" Nov 02 09:43:39.067783 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:39.067748 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:39.257077 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:39.257041 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:40.751926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:40.751870 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:40.940378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:40.940340 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:43:51.625268 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.625236 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z"] Nov 02 09:43:51.630816 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.630793 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.634859 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.634834 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 09:43:51.635065 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.635047 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b2fjp\"" Nov 02 09:43:51.641812 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.641788 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z"] Nov 02 09:43:51.749806 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.749768 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.750027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.749818 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.750027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.749835 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.750027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.749965 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.750201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.750025 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6wz\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-kube-api-access-tw6wz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.750201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.750087 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.750201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.750180 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.750348 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.750229 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.750348 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.750270 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851300 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851262 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851300 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851301 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6wz\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-kube-api-access-tw6wz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851352 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851397 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851436 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851467 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851521 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851572 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851600 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.851832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851815 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.852022 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.851964 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.852083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.852033 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.852127 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.852092 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.852316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.852299 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.853670 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.853653 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.853817 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.853801 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.864022 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.863996 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.864133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.864090 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6wz\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-kube-api-access-tw6wz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5qw4z\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:51.946312 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:51.946221 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:52.226117 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:52.226093 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z"] Nov 02 09:43:52.227618 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:43:52.227591 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d736f29_0f60_4d45_be3e_0ae7ef4efac5.slice/crio-b8ab137854cf5774be1f624faece2673b6921c00586dc623a89b7f213fdafa21 WatchSource:0}: Error finding container b8ab137854cf5774be1f624faece2673b6921c00586dc623a89b7f213fdafa21: Status 404 returned error can't find the container with id b8ab137854cf5774be1f624faece2673b6921c00586dc623a89b7f213fdafa21 Nov 02 09:43:52.627873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:52.627836 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" event={"ID":"7d736f29-0f60-4d45-be3e-0ae7ef4efac5","Type":"ContainerStarted","Data":"b8ab137854cf5774be1f624faece2673b6921c00586dc623a89b7f213fdafa21"} Nov 02 09:43:56.465686 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:56.465657 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:43:56.465991 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:56.465724 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:43:56.465991 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:56.465754 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:43:56.649403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:56.649369 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" event={"ID":"7d736f29-0f60-4d45-be3e-0ae7ef4efac5","Type":"ContainerStarted","Data":"3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82"} Nov 02 09:43:56.854279 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:43:56.854246 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:43:56.949006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:56.948969 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:43:57.948157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:57.948119 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 09:43:57.948608 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:57.948178 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 09:43:58.946931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:58.946858 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 09:43:58.947137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:58.946988 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 09:43:59.947368 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:59.947328 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 09:43:59.947754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:43:59.947392 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 09:44:00.946761 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:00.946729 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 09:44:00.946968 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:00.946777 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 09:44:01.947224 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:01.947186 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 09:44:01.947620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:01.947245 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 09:44:02.946539 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:02.946505 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 09:44:02.946710 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:02.946563 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 09:44:03.947231 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:03.947198 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 09:44:03.947623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:03.947248 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 09:44:05.659242 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:05.659198 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:44:05.659723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:05.659568 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:44:05.660537 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:05.660516 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 09:44:05.690165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:05.690108 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podStartSLOduration=10.45468527 podStartE2EDuration="14.690087865s" totalImagesPullingTime="4.235402595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:56.675030956 +0000 UTC m=+515.514972950" watchObservedRunningTime="2026-11-02 09:44:05.690087865 +0000 UTC m=+524.530030000" Nov 02 09:44:06.665736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.665701 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b"] Nov 02 09:44:06.669454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.669433 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.685361 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685332 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.685482 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685374 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.685740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685530 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.685740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685579 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcpng\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-kube-api-access-zcpng\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.685740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685609 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.685740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685631 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.685740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685688 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.685740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685738 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.686127 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.685779 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.687765 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.687745 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b"] Nov 02 09:44:06.717275 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.717246 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt"] Nov 02 09:44:06.721219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.721200 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.741331 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.741254 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt"] Nov 02 09:44:06.782362 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.782333 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:06.786840 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.786809 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.786952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.786852 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.786952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.786940 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.787053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.786968 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787003 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b6ee6-7885-4ac3-b559-3736307901b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.787155 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787117 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787155 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787150 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787260 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787164 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.787260 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787185 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcpng\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-kube-api-access-zcpng\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787387 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b6ee6-7885-4ac3-b559-3736307901b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.787556 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787533 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787562 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787586 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.787648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787485 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787617 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787688 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.787815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787721 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.787815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787771 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.788206 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.787980 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.788206 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.788182 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.788315 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.788226 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.788443 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.788415 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.788551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.788535 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg56n\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-kube-api-access-dg56n\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.789594 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.789575 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.790225 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.790207 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.799932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.799911 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcpng\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-kube-api-access-zcpng\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.803764 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.803739 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6pk2b\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:06.889307 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889283 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889321 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b6ee6-7885-4ac3-b559-3736307901b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889342 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889544 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889466 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889544 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889532 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889641 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889604 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg56n\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-kube-api-access-dg56n\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889700 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889644 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889700 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889667 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889700 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889682 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889863 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889746 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b6ee6-7885-4ac3-b559-3736307901b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.889947 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.889874 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.890057 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.890026 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.890111 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.890060 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.890405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.890388 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b6ee6-7885-4ac3-b559-3736307901b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.891462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.891440 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.891693 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.891674 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b6ee6-7885-4ac3-b559-3736307901b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.901778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.901759 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.902628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.902608 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg56n\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-kube-api-access-dg56n\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qtkkt\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:06.981223 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.981140 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:06.982053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:06.982036 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:07.037266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.037070 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:07.280190 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.280162 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b"] Nov 02 09:44:07.282443 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:44:07.282418 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc586fd_8e80_41bf_8d00_5f83e36107f7.slice/crio-9ec5d2e5b7dbfa79e01e79ba09c5409ecc5a64bfb0be1507060794238d284da2 WatchSource:0}: Error finding container 9ec5d2e5b7dbfa79e01e79ba09c5409ecc5a64bfb0be1507060794238d284da2: Status 404 returned error can't find the container with id 9ec5d2e5b7dbfa79e01e79ba09c5409ecc5a64bfb0be1507060794238d284da2 Nov 02 09:44:07.291339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.291307 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:07.291442 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.291373 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:07.291442 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.291406 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:07.296976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.296953 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:07.360617 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.360595 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt"] Nov 02 09:44:07.362977 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:44:07.362950 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507b6ee6_7885_4ac3_b559_3736307901b2.slice/crio-165864375e0031b552d9e89477b6493a9d778fbf65276923c261c23a150573b1 WatchSource:0}: Error finding container 165864375e0031b552d9e89477b6493a9d778fbf65276923c261c23a150573b1: Status 404 returned error can't find the container with id 165864375e0031b552d9e89477b6493a9d778fbf65276923c261c23a150573b1 Nov 02 09:44:07.366595 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.366564 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:07.366723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.366628 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:07.366723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.366672 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:07.490819 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.490785 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:07.673754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.673707 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:07.705450 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.705229 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" event={"ID":"507b6ee6-7885-4ac3-b559-3736307901b2","Type":"ContainerStarted","Data":"f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c"} Nov 02 09:44:07.705450 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.705288 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" event={"ID":"507b6ee6-7885-4ac3-b559-3736307901b2","Type":"ContainerStarted","Data":"165864375e0031b552d9e89477b6493a9d778fbf65276923c261c23a150573b1"} Nov 02 09:44:07.707155 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.707123 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" event={"ID":"4bc586fd-8e80-41bf-8d00-5f83e36107f7","Type":"ContainerStarted","Data":"3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214"} Nov 02 09:44:07.707296 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.707173 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" event={"ID":"4bc586fd-8e80-41bf-8d00-5f83e36107f7","Type":"ContainerStarted","Data":"9ec5d2e5b7dbfa79e01e79ba09c5409ecc5a64bfb0be1507060794238d284da2"} Nov 02 09:44:07.769500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.769438 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podStartSLOduration=1.769417848 podStartE2EDuration="1.769417848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:07.733722053 +0000 UTC m=+526.573664050" watchObservedRunningTime="2026-11-02 09:44:07.769417848 +0000 UTC m=+526.609359844" Nov 02 09:44:07.770732 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.770681 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podStartSLOduration=1.770669822 podStartE2EDuration="1.770669822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:07.766003236 +0000 UTC m=+526.605945230" watchObservedRunningTime="2026-11-02 09:44:07.770669822 +0000 UTC m=+526.610611817" Nov 02 09:44:07.861881 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.861844 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:07.988968 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:07.985972 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:08.027530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.027481 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6"] Nov 02 09:44:08.063176 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.063142 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 09:44:08.063425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.063395 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 09:44:08.063988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.063949 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:08.064168 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.064156 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6"] Nov 02 09:44:08.064325 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.064080 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.080971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.080938 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-000ffb-openshift-default-dockercfg-hz5bc\"" Nov 02 09:44:08.151915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.138696 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.152252 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.152216 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.152503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.152384 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.152503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.152451 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.152630 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.152544 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55w9\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-kube-api-access-r55w9\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.154366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.153149 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dea2d8-48a7-4f49-be4c-82d2f607554d-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.154366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.153312 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.154366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.153373 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.154366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.153424 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.203912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.198451 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:08.254957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.254591 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.254957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.254670 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.254957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.254709 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.254957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.254804 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.254957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.254843 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.254957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.254921 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.255407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.254979 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.255407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.255009 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55w9\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-kube-api-access-r55w9\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.255407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.255012 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.255407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.255050 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dea2d8-48a7-4f49-be4c-82d2f607554d-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.255407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.255161 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.255601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.255434 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.255854 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.255827 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.256020 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.255963 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dea2d8-48a7-4f49-be4c-82d2f607554d-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.258137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.257787 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.258137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.258071 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.269849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.269761 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.270005 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.269965 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55w9\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-kube-api-access-r55w9\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.416914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.416097 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:08.656795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.656754 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:08.824915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.813810 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6"] Nov 02 09:44:08.859453 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.859412 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:08.859652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.859633 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:08.859770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:08.859754 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:09.017928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.016387 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 09:44:09.017928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.016477 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 09:44:09.041139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.038024 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 09:44:09.041139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.038073 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 09:44:09.754230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.754191 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" event={"ID":"76dea2d8-48a7-4f49-be4c-82d2f607554d","Type":"ContainerStarted","Data":"f2ee28c74714430b486edf1a33eba85556d9fa8e6f11b11f47510d8309fd2859"} Nov 02 09:44:09.754433 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.754261 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" event={"ID":"76dea2d8-48a7-4f49-be4c-82d2f607554d","Type":"ContainerStarted","Data":"a7f2f79eb0ec53d5adf4dfebf5c17c1486044a95c066aa39c721448ebaa07d23"} Nov 02 09:44:09.779855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.779469 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" podStartSLOduration=1.779451227 podStartE2EDuration="1.779451227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:09.778422631 +0000 UTC m=+528.618364628" watchObservedRunningTime="2026-11-02 09:44:09.779451227 +0000 UTC m=+528.619393228" Nov 02 09:44:09.988917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.986296 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 09:44:09.988917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:09.986362 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 09:44:10.039965 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.039015 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 09:44:10.039965 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.039094 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 09:44:10.420325 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.420292 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:10.441935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.441879 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:10.512928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.511574 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:10.742511 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.742413 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:10.762532 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.762480 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:10.763698 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.763679 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:10.951727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.951690 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:10.985312 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.985275 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 09:44:10.985565 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:10.985536 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 09:44:11.038341 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.038256 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 09:44:11.038905 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.038848 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 09:44:11.931917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.931045 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:11.967345 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.967308 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj"] Nov 02 09:44:11.970909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.970868 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:11.979296 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.979270 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a110ef\"" Nov 02 09:44:11.979843 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.979822 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a110ef-dockercfg-ncfwk\"" Nov 02 09:44:11.991422 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.982601 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj"] Nov 02 09:44:11.994451 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:11.993155 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 09:44:12.016914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.005994 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 09:44:12.057297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.056958 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 09:44:12.057297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.057032 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 09:44:12.069458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.069204 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jks68\" (UniqueName: \"kubernetes.io/projected/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-kube-api-access-jks68\") pod \"payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj\" (UID: \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:12.069458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.069269 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj\" (UID: \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:12.147183 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.147125 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:12.170140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.170108 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks68\" (UniqueName: \"kubernetes.io/projected/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-kube-api-access-jks68\") pod \"payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj\" (UID: \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:12.170304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.170155 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj\" (UID: \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:12.171009 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.170990 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj\" (UID: \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:12.198565 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.186554 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jks68\" (UniqueName: \"kubernetes.io/projected/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-kube-api-access-jks68\") pod \"payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj\" (UID: \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:12.356722 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.356686 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:12.436914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.436340 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:12.575091 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.575060 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf"] Nov 02 09:44:12.579633 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.579608 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.618945 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.612811 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-a110ef-serving-cert\"" Nov 02 09:44:12.650914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.647713 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf"] Nov 02 09:44:12.654554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.652850 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:12.680583 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.679821 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.680583 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.679941 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8534bd6b-43ba-4033-b0e2-051a5f53eb04-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.680583 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.679970 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-metrics-certs\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.680583 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.680013 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xww25\" (UniqueName: \"kubernetes.io/projected/8534bd6b-43ba-4033-b0e2-051a5f53eb04-kube-api-access-xww25\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.715927 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.706375 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj"] Nov 02 09:44:12.780635 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.780604 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" event={"ID":"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d","Type":"ContainerStarted","Data":"5b46c97bbab69f4a7eb641cceef36e4bd968d485a644516ebb3284d6f194d20e"} Nov 02 09:44:12.781107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.781085 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.781207 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.781155 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8534bd6b-43ba-4033-b0e2-051a5f53eb04-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.781207 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.781181 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-metrics-certs\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.781284 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.781218 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xww25\" (UniqueName: \"kubernetes.io/projected/8534bd6b-43ba-4033-b0e2-051a5f53eb04-kube-api-access-xww25\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.781284 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:44:12.781267 4467 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-serving-cert: secret "maas-api-e2e-worker-w3-a110ef-serving-cert" not found Nov 02 09:44:12.781360 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:44:12.781332 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls podName:8534bd6b-43ba-4033-b0e2-051a5f53eb04 nodeName:}" failed. No retries permitted until 2026-11-02 09:44:13.281313253 +0000 UTC m=+532.121255243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls") pod "maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" (UID: "8534bd6b-43ba-4033-b0e2-051a5f53eb04") : secret "maas-api-e2e-worker-w3-a110ef-serving-cert" not found Nov 02 09:44:12.785911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.782106 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8534bd6b-43ba-4033-b0e2-051a5f53eb04-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.839912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.833687 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-metrics-certs\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.839912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.836236 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xww25\" (UniqueName: \"kubernetes.io/projected/8534bd6b-43ba-4033-b0e2-051a5f53eb04-kube-api-access-xww25\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:12.860539 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.858714 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:12.986388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.986277 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 09:44:12.986388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:12.986331 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 09:44:13.044027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.038997 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 09:44:13.044027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.039057 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 09:44:13.121217 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.121091 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:13.312434 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.312403 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:13.327478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.327425 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:13.335970 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.334704 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:13.559391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.559352 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:13.787842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.787774 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" event={"ID":"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d","Type":"ContainerStarted","Data":"017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412"} Nov 02 09:44:13.788110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.788011 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:13.843846 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.840780 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" podStartSLOduration=2.840759607 podStartE2EDuration="2.840759607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:13.837795516 +0000 UTC m=+532.677737511" watchObservedRunningTime="2026-11-02 09:44:13.840759607 +0000 UTC m=+532.680701604" Nov 02 09:44:13.865459 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.865431 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf"] Nov 02 09:44:13.868258 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:44:13.868223 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8534bd6b_43ba_4033_b0e2_051a5f53eb04.slice/crio-070bbb27d69ab8943b2945fe33b7fa038fc5bb1b73372f5b0b20b873c9a57cc4 WatchSource:0}: Error finding container 070bbb27d69ab8943b2945fe33b7fa038fc5bb1b73372f5b0b20b873c9a57cc4: Status 404 returned error can't find the container with id 070bbb27d69ab8943b2945fe33b7fa038fc5bb1b73372f5b0b20b873c9a57cc4 Nov 02 09:44:13.964875 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.964835 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq"] Nov 02 09:44:13.970869 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.970840 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:13.984544 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.984192 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 09:44:13.984544 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.984245 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 09:44:13.985260 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.985237 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq"] Nov 02 09:44:13.986877 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:13.986758 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2e80fb-serving-cert\"" Nov 02 09:44:14.038057 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.038018 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 09:44:14.038209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.038081 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 09:44:14.108495 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.108407 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:14.138585 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.138555 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.138585 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.138588 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgh7\" (UniqueName: \"kubernetes.io/projected/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-kube-api-access-tzgh7\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.139175 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.138626 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.139175 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.138790 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-metrics-certs\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.240107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.240071 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-metrics-certs\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.240297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.240119 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.240297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.240138 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgh7\" (UniqueName: \"kubernetes.io/projected/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-kube-api-access-tzgh7\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.240297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.240180 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.241064 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.241029 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.242772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.242750 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.242935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.242772 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-metrics-certs\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.254315 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.254285 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgh7\" (UniqueName: \"kubernetes.io/projected/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-kube-api-access-tzgh7\") pod \"maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.301260 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.301227 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:14.325643 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.325606 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:14.524753 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.524660 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:14.612180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.612144 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq"] Nov 02 09:44:14.797121 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.797024 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" event={"ID":"62503db2-d48c-47b3-bf5c-2ff6d65cc65b","Type":"ContainerStarted","Data":"4d3ee647c20cb5403f6e02891ef374ef8f34945f68f2fcfe89d8ea5c7bbab589"} Nov 02 09:44:14.800815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.800780 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" event={"ID":"8534bd6b-43ba-4033-b0e2-051a5f53eb04","Type":"ContainerStarted","Data":"4bef1b4449d2e992ff9238996085fbe48c704772cbf32cd564cd323e9f1921d6"} Nov 02 09:44:14.800996 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.800837 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" event={"ID":"8534bd6b-43ba-4033-b0e2-051a5f53eb04","Type":"ContainerStarted","Data":"070bbb27d69ab8943b2945fe33b7fa038fc5bb1b73372f5b0b20b873c9a57cc4"} Nov 02 09:44:14.800996 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.800881 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:14.825248 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.825195 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" podStartSLOduration=2.302366234 podStartE2EDuration="2.825181656s" totalImagesPullingTime="522.815422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:14.823181528 +0000 UTC m=+533.663123537" watchObservedRunningTime="2026-11-02 09:44:14.825181656 +0000 UTC m=+533.665123650" Nov 02 09:44:14.982463 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.982426 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 09:44:14.982616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:14.982493 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 09:44:15.038495 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:15.038460 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 09:44:15.038671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:15.038526 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 09:44:15.143043 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:15.142998 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:15.809173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:15.809121 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" event={"ID":"62503db2-d48c-47b3-bf5c-2ff6d65cc65b","Type":"ContainerStarted","Data":"2fdb279f03ce351211ac8ea752d561092fe7ce7c8346a3f851f78461d53fa99d"} Nov 02 09:44:15.809380 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:15.809208 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:15.835409 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:15.835347 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" podStartSLOduration=2.4202751989999998 podStartE2EDuration="2.835333848s" totalImagesPullingTime="415.058649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:15.833397474 +0000 UTC m=+534.673339469" watchObservedRunningTime="2026-11-02 09:44:15.835333848 +0000 UTC m=+534.675275842" Nov 02 09:44:15.998915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:15.998301 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 09:44:15.998915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:15.998360 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 09:44:16.009925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.009143 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr"] Nov 02 09:44:16.016189 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.014053 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.040153 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.038373 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr"] Nov 02 09:44:16.040153 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.038613 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-000ffb-serving-cert\"" Nov 02 09:44:16.137968 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.137324 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:44:16.156508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.144383 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:16.169909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.167625 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.169909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.167699 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-metrics-certs\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.169909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.167739 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8l8\" (UniqueName: \"kubernetes.io/projected/02c00d49-a410-4910-a8ac-e7c489ce1ef8-kube-api-access-2m8l8\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.169909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.167800 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c00d49-a410-4910-a8ac-e7c489ce1ef8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.269417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.269371 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c00d49-a410-4910-a8ac-e7c489ce1ef8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.269622 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.269551 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.269694 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.269661 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-metrics-certs\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.269756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.269719 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8l8\" (UniqueName: \"kubernetes.io/projected/02c00d49-a410-4910-a8ac-e7c489ce1ef8-kube-api-access-2m8l8\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.270698 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.270670 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c00d49-a410-4910-a8ac-e7c489ce1ef8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.273791 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.273638 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.274524 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.274499 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-metrics-certs\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.287855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.282487 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8l8\" (UniqueName: \"kubernetes.io/projected/02c00d49-a410-4910-a8ac-e7c489ce1ef8-kube-api-access-2m8l8\") pod \"maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.353022 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.352990 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:16.354055 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.354023 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:16.540052 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.540013 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:16.710323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.710234 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr"] Nov 02 09:44:16.821911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:16.819408 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" event={"ID":"02c00d49-a410-4910-a8ac-e7c489ce1ef8","Type":"ContainerStarted","Data":"029cfa185f566c8e2db9f655a07df9797a69e75214b3c6f6c7899d53ec157e17"} Nov 02 09:44:17.081130 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.081076 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:44:17.081327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.081157 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:44:17.197857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.197682 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:17.549802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.549768 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:17.551212 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.551153 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:17.553554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.553507 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 09:44:17.645988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.644174 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:17.645988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.644885 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:17.649647 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.649614 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:44:17.825332 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.825291 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" event={"ID":"02c00d49-a410-4910-a8ac-e7c489ce1ef8","Type":"ContainerStarted","Data":"194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa"} Nov 02 09:44:17.825530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.825516 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:17.846481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.846434 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" podStartSLOduration=2.34920425 podStartE2EDuration="2.846419693s" totalImagesPullingTime="497.215443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:17.845164091 +0000 UTC m=+536.685106088" watchObservedRunningTime="2026-11-02 09:44:17.846419693 +0000 UTC m=+536.686361721" Nov 02 09:44:17.974798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:17.974718 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:18.683853 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:18.683813 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:19.463637 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:19.463607 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:19.839240 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:19.839203 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:44:20.381059 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:20.381032 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:21.101866 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:21.101827 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:21.825710 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:21.825665 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 09:44:21.826169 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:21.826144 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:44:21.826306 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:21.826277 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:22.569617 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:22.569588 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:23.512821 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:23.512788 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:23.837440 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:23.837408 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:44:24.335728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:24.335697 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:24.807798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:24.807769 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 09:44:25.244584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:25.244502 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:25.648680 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:25.648649 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:25.826709 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:25.826680 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:26.189649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:26.189614 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:26.396482 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:26.396451 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:26.608141 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:26.608104 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:26.855395 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:44:26.855355 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:44:27.545094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:27.545059 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:27.740323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:27.740288 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:28.470772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:28.470736 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:29.165217 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:29.165183 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:44:29.269858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:29.269773 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:29.888928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:29.888763 4467 generic.go:356] "Generic (PLEG): container finished" podID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerID="f2ee28c74714430b486edf1a33eba85556d9fa8e6f11b11f47510d8309fd2859" exitCode=0 Nov 02 09:44:29.888928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:29.888842 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" event={"ID":"76dea2d8-48a7-4f49-be4c-82d2f607554d","Type":"ContainerDied","Data":"f2ee28c74714430b486edf1a33eba85556d9fa8e6f11b11f47510d8309fd2859"} Nov 02 09:44:29.889385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:29.889364 4467 scope.go:122] "RemoveContainer" containerID="f2ee28c74714430b486edf1a33eba85556d9fa8e6f11b11f47510d8309fd2859" Nov 02 09:44:29.890997 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:29.890964 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:29.891090 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:29.891056 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:29.891146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:29.891107 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:30.381096 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:30.381047 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:30.911913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:30.910950 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" event={"ID":"76dea2d8-48a7-4f49-be4c-82d2f607554d","Type":"ContainerStarted","Data":"54b76473dec66b8f44d0cfb46747ee011ab935cd829a635a5905a59d29caf97a"} Nov 02 09:44:31.210977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:31.210399 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:31.424354 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:31.424316 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:31.425087 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:31.424994 4467 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 09:44:31.425303 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:31.425274 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 09:44:32.412915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:32.412556 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:32.423916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:32.421376 4467 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 09:44:32.423916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:32.421456 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 09:44:32.672659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:32.672565 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:33.418715 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:33.418260 4467 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 09:44:33.418715 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:33.418328 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 09:44:33.593952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:33.593384 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:44:33.593952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:33.593452 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:44:33.695826 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:33.695729 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:33.894776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:33.894740 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:34.039978 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.029441 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm"] Nov 02 09:44:34.099202 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.096246 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.119358 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.119311 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm"] Nov 02 09:44:34.123916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.123870 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d2e277-openshift-default-dockercfg-mnb9x\"" Nov 02 09:44:34.160855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.160812 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187452 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-token\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187530 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcvp\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-kube-api-access-xwcvp\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187572 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-podinfo\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187641 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-certs\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187679 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2333ec-e848-48dd-a738-039a7dd0a74f-istiod-ca-cert\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187710 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-socket\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187766 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-data\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187802 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-credential-socket\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.199914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.187850 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-envoy\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288453 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-data\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288511 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-credential-socket\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288553 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-envoy\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288605 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-token\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288654 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcvp\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-kube-api-access-xwcvp\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288688 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-podinfo\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288758 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-certs\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288792 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2333ec-e848-48dd-a738-039a7dd0a74f-istiod-ca-cert\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.288821 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-socket\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.289289 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-socket\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.289518 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-data\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.289729 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-credential-socket\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.305784 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-certs\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.311958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.306400 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2333ec-e848-48dd-a738-039a7dd0a74f-istiod-ca-cert\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.339506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.338417 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-podinfo\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.339506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.339048 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcvp\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-kube-api-access-xwcvp\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.339506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.339153 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-envoy\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.339506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.339421 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-token\") pod \"e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.422388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.422353 4467 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 09:44:34.422701 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.422658 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 09:44:34.472405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.472376 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:34.571112 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.571079 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:34.768675 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.768641 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:34.815833 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.804699 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm"] Nov 02 09:44:34.820253 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:44:34.820184 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2333ec_e848_48dd_a738_039a7dd0a74f.slice/crio-4741f17b7bb7e065dd0f08b6ebdac929052085e8ea4bd52e34a0cd814a82ada2 WatchSource:0}: Error finding container 4741f17b7bb7e065dd0f08b6ebdac929052085e8ea4bd52e34a0cd814a82ada2: Status 404 returned error can't find the container with id 4741f17b7bb7e065dd0f08b6ebdac929052085e8ea4bd52e34a0cd814a82ada2 Nov 02 09:44:34.822563 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.822532 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:34.822639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.822602 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:34.822639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.822628 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:34.971161 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.971130 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:34.973547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.973520 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" event={"ID":"cc2333ec-e848-48dd-a738-039a7dd0a74f","Type":"ContainerStarted","Data":"0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e"} Nov 02 09:44:34.973693 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:34.973573 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" event={"ID":"cc2333ec-e848-48dd-a738-039a7dd0a74f","Type":"ContainerStarted","Data":"4741f17b7bb7e065dd0f08b6ebdac929052085e8ea4bd52e34a0cd814a82ada2"} Nov 02 09:44:35.423445 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:35.423001 4467 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 09:44:35.423445 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:35.423064 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 09:44:35.473113 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:35.473061 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:35.479026 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:35.478999 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:35.533918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:35.523112 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" podStartSLOduration=2.523092583 podStartE2EDuration="2.523092583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:35.0010473 +0000 UTC m=+553.840989323" watchObservedRunningTime="2026-11-02 09:44:35.523092583 +0000 UTC m=+554.363034578" Nov 02 09:44:35.772294 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:35.772192 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:35.980766 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:35.980742 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:35.981836 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:35.981818 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:36.418921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:36.418293 4467 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 09:44:36.418921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:36.418356 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 09:44:36.853913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:36.850691 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:37.400184 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:37.400148 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:37.439236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:37.439206 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:37.440843 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:37.440818 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:37.448107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:37.448085 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:44:37.667963 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:37.666247 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:38.331913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:38.331777 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:38.548407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:38.548371 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:44:38.875913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:38.859029 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:39.092337 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:39.092285 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:39.656864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:39.656830 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:40.541941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:40.541875 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:41.206610 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.206578 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs"] Nov 02 09:44:41.212056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.212028 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.222797 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.222734 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d2e277-serving-cert\"" Nov 02 09:44:41.227064 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.227038 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs"] Nov 02 09:44:41.317802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.317770 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsx4b\" (UniqueName: \"kubernetes.io/projected/929261f1-a83d-4e36-98f8-f9f4cec09ef9-kube-api-access-xsx4b\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.318006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.317827 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.318006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.317959 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929261f1-a83d-4e36-98f8-f9f4cec09ef9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.318006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.317981 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-metrics-certs\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.377525 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.377495 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:41.418707 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.418673 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsx4b\" (UniqueName: \"kubernetes.io/projected/929261f1-a83d-4e36-98f8-f9f4cec09ef9-kube-api-access-xsx4b\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.418950 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.418729 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.418950 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.418756 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929261f1-a83d-4e36-98f8-f9f4cec09ef9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.418950 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.418777 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-metrics-certs\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.419621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.419601 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929261f1-a83d-4e36-98f8-f9f4cec09ef9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.421380 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.421357 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.421468 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.421361 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-metrics-certs\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.431407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.431385 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsx4b\" (UniqueName: \"kubernetes.io/projected/929261f1-a83d-4e36-98f8-f9f4cec09ef9-kube-api-access-xsx4b\") pod \"maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.532969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.532848 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:41.575287 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.575247 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:41.859316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:41.859282 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs"] Nov 02 09:44:41.861220 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:44:41.861190 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929261f1_a83d_4e36_98f8_f9f4cec09ef9.slice/crio-c672765614bfbcd86fa03457b1e67de76c298b4e3de1b6d5587c4a2c5d191a30 WatchSource:0}: Error finding container c672765614bfbcd86fa03457b1e67de76c298b4e3de1b6d5587c4a2c5d191a30: Status 404 returned error can't find the container with id c672765614bfbcd86fa03457b1e67de76c298b4e3de1b6d5587c4a2c5d191a30 Nov 02 09:44:42.093226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:42.093185 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" event={"ID":"929261f1-a83d-4e36-98f8-f9f4cec09ef9","Type":"ContainerStarted","Data":"c672765614bfbcd86fa03457b1e67de76c298b4e3de1b6d5587c4a2c5d191a30"} Nov 02 09:44:43.100541 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:43.100508 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" event={"ID":"929261f1-a83d-4e36-98f8-f9f4cec09ef9","Type":"ContainerStarted","Data":"1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4"} Nov 02 09:44:43.101069 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:43.100636 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:43.291037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:43.291002 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:44.163747 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:44.163717 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:44.754676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:44.754642 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:44.963672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:44.963640 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:45.147085 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:45.147052 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:45.473633 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:45.473537 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:46.167767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:46.167736 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:46.903845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:46.903804 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:47.255710 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:47.255608 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" podStartSLOduration=5.843940221 podStartE2EDuration="6.255591179s" totalImagesPullingTime="411.650958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:43.121586485 +0000 UTC m=+561.961528480" watchObservedRunningTime="2026-11-02 09:44:47.255591179 +0000 UTC m=+566.095533199" Nov 02 09:44:47.257042 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:47.257021 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:44:47.644532 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:47.644496 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:48.601916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:48.589403 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:49.116023 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:49.115990 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:44:49.356278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:49.356215 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:50.145354 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:50.145323 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:50.844512 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:50.844482 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:51.543512 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:51.543478 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:51.723382 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:51.723348 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:52.135916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:52.124570 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:52.314010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:52.313974 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:52.547304 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:52.547214 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:53.324385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:53.324332 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:54.194580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:54.194497 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:54.673646 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:54.673606 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:55.043215 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:55.043128 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:55.239176 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:55.239145 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:55.415033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:55.414994 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:55.867499 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:55.867462 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:56.478433 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:56.478398 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:56.702277 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:56.702241 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:56.857369 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:44:56.857338 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:44:56.890517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:56.890030 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:56.915690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:56.915657 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:44:57.189423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:57.189344 4467 generic.go:356] "Generic (PLEG): container finished" podID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerID="0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e" exitCode=0 Nov 02 09:44:57.189582 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:57.189417 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" event={"ID":"cc2333ec-e848-48dd-a738-039a7dd0a74f","Type":"ContainerDied","Data":"0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e"} Nov 02 09:44:57.189871 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:57.189850 4467 scope.go:122] "RemoveContainer" containerID="0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e" Nov 02 09:44:57.191200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:57.191173 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:57.191255 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:57.191241 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:57.191291 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:57.191273 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:44:57.768036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:57.768003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:58.210225 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.210186 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" event={"ID":"cc2333ec-e848-48dd-a738-039a7dd0a74f","Type":"ContainerStarted","Data":"0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4"} Nov 02 09:44:58.427916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.426464 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb"] Nov 02 09:44:58.438955 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.438430 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb"] Nov 02 09:44:58.438955 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.438520 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.444387 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.443598 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-bde944-serving-cert\"" Nov 02 09:44:58.473374 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.473302 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:44:58.473524 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.473503 4467 patch_prober.go:29] interesting pod/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 09:44:58.473598 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.473568 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 09:44:58.548181 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.548137 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:58.621222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.621177 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ee45a7-858e-4e54-93b9-acb0f12be41c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.621432 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.621406 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.621569 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.621549 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7vb\" (UniqueName: \"kubernetes.io/projected/90ee45a7-858e-4e54-93b9-acb0f12be41c-kube-api-access-cv7vb\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.621659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.621617 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.722609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.722560 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.722779 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.722660 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7vb\" (UniqueName: \"kubernetes.io/projected/90ee45a7-858e-4e54-93b9-acb0f12be41c-kube-api-access-cv7vb\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.722779 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.722706 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.722779 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.722750 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ee45a7-858e-4e54-93b9-acb0f12be41c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.723667 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.723608 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ee45a7-858e-4e54-93b9-acb0f12be41c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.725799 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.725769 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.725883 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.725811 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.735460 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.735436 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7vb\" (UniqueName: \"kubernetes.io/projected/90ee45a7-858e-4e54-93b9-acb0f12be41c-kube-api-access-cv7vb\") pod \"maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:58.735999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.735969 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:58.759140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:58.759107 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:44:59.078315 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:59.078293 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb"] Nov 02 09:44:59.080028 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:44:59.080001 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ee45a7_858e_4e54_93b9_acb0f12be41c.slice/crio-331bfbd3f16d35fe256c606fdbf3b4323af967d4a05d33cedacd3cfd2e48ae07 WatchSource:0}: Error finding container 331bfbd3f16d35fe256c606fdbf3b4323af967d4a05d33cedacd3cfd2e48ae07: Status 404 returned error can't find the container with id 331bfbd3f16d35fe256c606fdbf3b4323af967d4a05d33cedacd3cfd2e48ae07 Nov 02 09:44:59.227137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:59.226528 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" event={"ID":"90ee45a7-858e-4e54-93b9-acb0f12be41c","Type":"ContainerStarted","Data":"331bfbd3f16d35fe256c606fdbf3b4323af967d4a05d33cedacd3cfd2e48ae07"} Nov 02 09:44:59.357798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:59.357721 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:44:59.473180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:59.473141 4467 patch_prober.go:29] interesting pod/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 09:44:59.473351 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:44:59.473200 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 09:45:00.240393 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:00.240346 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" event={"ID":"90ee45a7-858e-4e54-93b9-acb0f12be41c","Type":"ContainerStarted","Data":"2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e"} Nov 02 09:45:00.241005 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:00.240511 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:45:00.259256 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:00.259228 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:00.263404 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:00.263360 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" podStartSLOduration=1.795275044 podStartE2EDuration="2.263345678s" totalImagesPullingTime="468.070634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:00.262441453 +0000 UTC m=+579.102383448" watchObservedRunningTime="2026-11-02 09:45:00.263345678 +0000 UTC m=+579.103287673" Nov 02 09:45:00.473522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:00.473486 4467 patch_prober.go:29] interesting pod/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 09:45:00.473778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:00.473754 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 09:45:00.952723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:00.952684 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:01.472978 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:01.472941 4467 patch_prober.go:29] interesting pod/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 09:45:01.473373 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:01.473009 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 09:45:01.663883 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:01.663851 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:01.872228 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:01.872195 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:02.473109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:02.473069 4467 patch_prober.go:29] interesting pod/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 09:45:02.473498 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:02.473151 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 09:45:02.597469 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:02.597403 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:03.366375 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:03.366292 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:03.477231 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:03.477201 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:45:03.477628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:03.477466 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:45:03.478325 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:03.478308 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:45:04.229303 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:04.229264 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:05.110277 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:05.110241 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:05.858772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:05.858723 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:06.251880 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:06.251808 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:45:06.746905 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:06.746852 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:07.749588 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:07.749560 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:07.977678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:07.977645 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:45:08.468758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:08.468723 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:09.597520 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:09.597485 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:10.180659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:10.180622 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:10.519237 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:10.519150 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:10.710788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:10.710749 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:10.921516 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:10.921483 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:11.163248 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:11.163204 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:11.367025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:11.366984 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:11.697360 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:11.697283 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:12.473582 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:12.473545 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:13.266028 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:13.265993 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:14.176355 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:14.176323 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:14.452429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:14.452401 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:14.673654 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:14.673620 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:15.343476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:15.343441 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:15.818119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:15.818088 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:16.062992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:16.062962 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:16.679685 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:16.679646 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:17.612858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:17.612824 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:18.479122 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:18.479084 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:19.108038 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.108003 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l"] Nov 02 09:45:19.129254 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.129226 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l"] Nov 02 09:45:19.129492 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.129318 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.136362 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.136332 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-9e7156-serving-cert\"" Nov 02 09:45:19.254613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.254582 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.254782 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.254660 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:19.254782 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.254708 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.254782 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.254779 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv25g\" (UniqueName: \"kubernetes.io/projected/7f08e396-bdfc-46c6-9403-1134d5750473-kube-api-access-bv25g\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.254914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.254809 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f08e396-bdfc-46c6-9403-1134d5750473-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.355975 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.355937 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f08e396-bdfc-46c6-9403-1134d5750473-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.356152 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.355996 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.356152 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.356056 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.356152 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.356119 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv25g\" (UniqueName: \"kubernetes.io/projected/7f08e396-bdfc-46c6-9403-1134d5750473-kube-api-access-bv25g\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.356651 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.356624 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f08e396-bdfc-46c6-9403-1134d5750473-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.358558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.358510 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.358683 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.358664 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.370353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.370326 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv25g\" (UniqueName: \"kubernetes.io/projected/7f08e396-bdfc-46c6-9403-1134d5750473-kube-api-access-bv25g\") pod \"maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.433332 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.433295 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:19.447669 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.447637 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:19.560594 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.560562 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:45:19.736713 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:19.736675 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l"] Nov 02 09:45:19.739371 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:45:19.739345 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f08e396_bdfc_46c6_9403_1134d5750473.slice/crio-6a56112dddcb1c400cb1198b198bc810a81a7d03fd774a9ed1006234059dd5dd WatchSource:0}: Error finding container 6a56112dddcb1c400cb1198b198bc810a81a7d03fd774a9ed1006234059dd5dd: Status 404 returned error can't find the container with id 6a56112dddcb1c400cb1198b198bc810a81a7d03fd774a9ed1006234059dd5dd Nov 02 09:45:20.160704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:20.160663 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:20.353605 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:20.353568 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" event={"ID":"7f08e396-bdfc-46c6-9403-1134d5750473","Type":"ContainerStarted","Data":"5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec"} Nov 02 09:45:20.353781 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:20.353621 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" event={"ID":"7f08e396-bdfc-46c6-9403-1134d5750473","Type":"ContainerStarted","Data":"6a56112dddcb1c400cb1198b198bc810a81a7d03fd774a9ed1006234059dd5dd"} Nov 02 09:45:20.353781 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:20.353685 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:20.396364 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:20.396295 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" podStartSLOduration=0.941717682 podStartE2EDuration="1.396276327s" totalImagesPullingTime="454.558645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:20.39537717 +0000 UTC m=+599.235319167" watchObservedRunningTime="2026-11-02 09:45:20.396276327 +0000 UTC m=+599.236218323" Nov 02 09:45:21.052818 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:21.052784 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:21.772159 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:21.772123 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:22.281968 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.281932 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg"] Nov 02 09:45:22.285814 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.285796 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.296570 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.296546 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a110ef-openshift-default-dockercfg-qjmqj\"" Nov 02 09:45:22.299389 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.299364 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg"] Nov 02 09:45:22.390177 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390140 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.390370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390206 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xz2q\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-kube-api-access-4xz2q\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.390370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390337 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.390457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390385 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.390457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390415 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.390576 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390561 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.390626 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390593 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.390685 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390624 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.390859 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.390833 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492414 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492382 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492414 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492417 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492675 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492440 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492675 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492604 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492675 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492663 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492877 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492715 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xz2q\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-kube-api-access-4xz2q\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492877 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492777 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492877 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492809 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.492877 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492860 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.493097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492875 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.493097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.492962 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.493311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.493293 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.493431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.493406 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.493592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.493575 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.495480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.495454 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.495764 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.495740 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.505063 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.505033 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.505830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.505764 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xz2q\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-kube-api-access-4xz2q\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.563783 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.563748 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:22.601968 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.601930 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:22.883705 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.883677 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg"] Nov 02 09:45:22.886406 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:45:22.886376 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac2dfd4_0623_4aed_b591_3f8334b74cc9.slice/crio-67c502f7f11499f6df1c757fbf455ca05cdcde7b7bc4eb5609c756dfd27f3c5a WatchSource:0}: Error finding container 67c502f7f11499f6df1c757fbf455ca05cdcde7b7bc4eb5609c756dfd27f3c5a: Status 404 returned error can't find the container with id 67c502f7f11499f6df1c757fbf455ca05cdcde7b7bc4eb5609c756dfd27f3c5a Nov 02 09:45:22.889245 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.889212 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:45:22.889342 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.889284 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:45:22.889342 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:22.889316 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:45:23.354595 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:23.354563 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:23.377614 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:23.377579 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" event={"ID":"bac2dfd4-0623-4aed-b591-3f8334b74cc9","Type":"ContainerStarted","Data":"744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b"} Nov 02 09:45:23.377799 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:23.377670 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" event={"ID":"bac2dfd4-0623-4aed-b591-3f8334b74cc9","Type":"ContainerStarted","Data":"67c502f7f11499f6df1c757fbf455ca05cdcde7b7bc4eb5609c756dfd27f3c5a"} Nov 02 09:45:23.404691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:23.404632 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podStartSLOduration=1.404619296 podStartE2EDuration="1.404619296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:23.401843626 +0000 UTC m=+602.241785620" watchObservedRunningTime="2026-11-02 09:45:23.404619296 +0000 UTC m=+602.244561291" Nov 02 09:45:23.605976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:23.603267 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:23.619916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:23.614049 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 09:45:23.619916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:23.614113 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 09:45:24.103915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:24.096267 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:24.604914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:24.604419 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 09:45:24.604914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:24.604484 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 09:45:25.145998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:25.145960 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:25.328221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:25.328188 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:25.603386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:25.603347 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 09:45:25.603682 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:25.603651 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 09:45:25.899191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:25.899096 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:26.365323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:26.365294 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:45:26.607324 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:26.607286 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 09:45:26.607525 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:26.607352 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 09:45:26.796972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:26.796351 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:26.854574 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:45:26.854534 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:45:27.605360 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:27.605326 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 09:45:27.605974 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:27.605943 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 09:45:27.615403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:27.615373 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:28.414226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:28.414148 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:28.603327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:28.603287 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 09:45:28.603627 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:28.603600 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 09:45:29.187169 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:29.187132 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:29.205696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:29.205650 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:45:29.603301 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:29.603259 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 09:45:29.603530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:29.603332 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 09:45:30.172310 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:30.172254 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:30.962634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:30.962599 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:31.644056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:31.644010 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:45:31.644389 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:31.644070 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:45:31.672834 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:31.672803 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:31.673285 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:31.673233 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:31.674147 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:31.674127 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 09:45:31.818278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:31.818244 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:32.660747 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:32.660715 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:33.450505 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:33.450466 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:34.160395 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:34.160352 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:34.516674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:34.516562 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:34.955424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:34.955392 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:35.720814 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:35.720771 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:35.978642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:35.978524 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:36.660611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:36.660577 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:37.400253 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.400213 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8"] Nov 02 09:45:37.405426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.405396 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.411881 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.411841 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2e80fb-openshift-default-dockercfg-545dt\"" Nov 02 09:45:37.427390 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.427351 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8"] Nov 02 09:45:37.448472 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.448426 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:37.573146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573108 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.573328 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573159 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.573328 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573249 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvm2\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-kube-api-access-8fvm2\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.573328 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573302 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.573438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573330 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.573438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573349 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.573438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573380 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.573532 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573478 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.573532 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.573509 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.674661 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674567 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.674661 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674603 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.674661 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674636 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674668 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674699 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674801 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674860 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674948 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvm2\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-kube-api-access-8fvm2\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.674981 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.675168 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.675366 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.675453 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675570 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.675549 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.675788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.675769 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.677124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.677091 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.678180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.678163 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.687098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.687064 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.688400 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.688372 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvm2\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-kube-api-access-8fvm2\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:37.731394 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:37.731360 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:38.014621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.014595 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8"] Nov 02 09:45:38.017061 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:45:38.017025 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78bbcb8_1829_4f6a_8738_9c69969fb66b.slice/crio-5875dcf2f0d4af5a5c0c580479794a504046a051b851a9544757a22f6fd3c3c8 WatchSource:0}: Error finding container 5875dcf2f0d4af5a5c0c580479794a504046a051b851a9544757a22f6fd3c3c8: Status 404 returned error can't find the container with id 5875dcf2f0d4af5a5c0c580479794a504046a051b851a9544757a22f6fd3c3c8 Nov 02 09:45:38.019610 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.019577 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:45:38.019719 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.019658 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:45:38.019783 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.019718 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:45:38.195442 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.195412 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:38.517581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.517542 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" event={"ID":"b78bbcb8-1829-4f6a-8738-9c69969fb66b","Type":"ContainerStarted","Data":"9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851"} Nov 02 09:45:38.518119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.517607 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" event={"ID":"b78bbcb8-1829-4f6a-8738-9c69969fb66b","Type":"ContainerStarted","Data":"5875dcf2f0d4af5a5c0c580479794a504046a051b851a9544757a22f6fd3c3c8"} Nov 02 09:45:38.543952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.543875 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podStartSLOduration=1.5438569279999999 podStartE2EDuration="1.543856928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:38.541809313 +0000 UTC m=+617.381751310" watchObservedRunningTime="2026-11-02 09:45:38.543856928 +0000 UTC m=+617.383798924" Nov 02 09:45:38.731597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.731552 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:38.943470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:38.943438 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:39.654499 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:39.654463 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:39.733854 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:39.733814 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 09:45:39.734057 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:39.733884 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 09:45:39.847555 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:39.847521 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:40.059504 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:40.059470 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:40.511976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:40.511209 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:40.731785 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:40.731753 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 09:45:40.732286 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:40.732261 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 09:45:41.009924 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:41.009038 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:45:41.392680 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:41.392570 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:41.732334 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:41.732248 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 09:45:41.732334 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:41.732301 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 09:45:42.361440 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:42.361406 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:42.731735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:42.731648 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 09:45:42.731735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:42.731701 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 09:45:42.918189 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:42.918147 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:43.312410 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:43.312376 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:43.732087 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:43.732003 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 09:45:43.732087 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:43.732069 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 09:45:44.065302 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:44.065267 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:44.771647 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:44.771610 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:45.139753 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:45.139714 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:45.595140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:45.595099 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:45.776064 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:45.776032 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:45:45.776243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:45.776109 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:45:45.942313 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:45.942232 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:45.942584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:45.942560 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:45.994910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:45.994858 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:45:46.374964 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:46.374930 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:47.190712 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:47.190667 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:48.225763 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:48.225725 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:49.048813 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:49.048778 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:49.752007 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:49.751975 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:50.453362 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:50.453327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:51.151748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:51.151710 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:51.963370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:51.963298 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:52.761156 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:52.761123 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:52.957495 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:52.957467 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:53.560526 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:53.560498 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:54.269977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:54.269947 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:45:54.432999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:54.432961 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:55.254414 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:55.254366 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:56.054423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:56.054395 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:56.786032 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:56.786000 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:56.856860 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:45:56.856824 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:45:57.572574 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:57.572541 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:58.347690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:58.347649 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:59.063436 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:59.063404 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:45:59.964567 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:45:59.964497 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:00.746555 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:00.746522 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:01.455336 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:01.455301 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:02.155110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:02.155077 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:02.854667 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:02.854634 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:03.289157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:03.288990 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:03.653767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:03.653733 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:04.453846 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:04.453815 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:05.259239 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:05.259209 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:05.856870 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:05.856832 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:46:05.947285 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:05.947245 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:06.741857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:06.741817 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:07.364394 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.364362 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r"] Nov 02 09:46:07.368344 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.368324 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.412751 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.412726 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r"] Nov 02 09:46:07.466275 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.466248 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:07.542370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542336 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6t2z\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-kube-api-access-q6t2z\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.542370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542375 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.542634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542503 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8396fdd2-c412-459d-956d-e0a707278dd5-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.542634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542564 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.542634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542622 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.542810 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542660 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.542810 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542751 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.542810 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542805 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8396fdd2-c412-459d-956d-e0a707278dd5-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.542999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.542867 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643562 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643466 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8396fdd2-c412-459d-956d-e0a707278dd5-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643562 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643535 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643580 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6t2z\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-kube-api-access-q6t2z\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643598 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643644 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8396fdd2-c412-459d-956d-e0a707278dd5-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643665 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643709 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643739 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.643809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.643779 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.644268 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.644213 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.644268 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.644233 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.644425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.644373 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.644496 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.644473 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.644649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.644632 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8396fdd2-c412-459d-956d-e0a707278dd5-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.646241 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.646208 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.646343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.646274 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:07.646343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.646295 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8396fdd2-c412-459d-956d-e0a707278dd5-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.658958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.658934 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.659135 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.659119 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6t2z\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-kube-api-access-q6t2z\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:07.711348 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:07.711314 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:08.085541 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:08.085511 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r"] Nov 02 09:46:08.087058 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:46:08.087034 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8396fdd2_c412_459d_956d_e0a707278dd5.slice/crio-c460b4133f24227116aa481c29f3c03f666733df7f03558194f7e2d411e4e78e WatchSource:0}: Error finding container c460b4133f24227116aa481c29f3c03f666733df7f03558194f7e2d411e4e78e: Status 404 returned error can't find the container with id c460b4133f24227116aa481c29f3c03f666733df7f03558194f7e2d411e4e78e Nov 02 09:46:08.090185 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:08.090143 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:08.090339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:08.090261 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:08.090422 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:08.090350 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:08.553489 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:08.553400 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:08.710919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:08.710837 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" event={"ID":"8396fdd2-c412-459d-956d-e0a707278dd5","Type":"ContainerStarted","Data":"ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048"} Nov 02 09:46:08.711128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:08.710929 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" event={"ID":"8396fdd2-c412-459d-956d-e0a707278dd5","Type":"ContainerStarted","Data":"c460b4133f24227116aa481c29f3c03f666733df7f03558194f7e2d411e4e78e"} Nov 02 09:46:08.718923 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:08.712027 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:09.508945 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:09.508479 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:09.715994 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:09.715956 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 09:46:09.716559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:09.716033 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 09:46:10.267157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:10.267112 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:10.556765 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:10.556690 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:10.712916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:10.712321 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 09:46:10.712916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:10.712390 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 09:46:11.059723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:11.059689 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:11.711806 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:11.711759 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 09:46:11.712053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:11.711823 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 09:46:11.862050 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:11.862018 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:12.711803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:12.711765 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 09:46:12.712586 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:12.712543 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 09:46:12.764468 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:12.764435 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:13.673469 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:13.673436 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:13.713393 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:13.713345 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 09:46:13.713948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:13.713417 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 09:46:14.508621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:14.508579 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:14.698747 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:14.698697 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:14.711836 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:14.711804 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 09:46:14.712002 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:14.711868 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 09:46:15.398397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:15.398350 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:16.146560 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:16.146531 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:16.152672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:16.152544 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:16.152672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:16.152634 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:46:16.177799 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:16.177750 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podStartSLOduration=9.177737593 podStartE2EDuration="9.177737593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:08.74004305 +0000 UTC m=+647.579985045" watchObservedRunningTime="2026-11-02 09:46:16.177737593 +0000 UTC m=+655.017679587" Nov 02 09:46:16.357122 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:16.357029 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:17.253032 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:17.253000 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:17.998011 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:17.997980 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:18.763786 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:18.763740 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:19.459701 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:19.459665 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:20.270426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:20.270392 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:20.275953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:20.275928 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:46:21.165705 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:21.164766 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:21.446975 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:21.442705 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:22.059054 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:22.059015 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:22.473722 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:22.473637 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:22.655048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:22.655011 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:23.136511 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:23.136471 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:23.554067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:23.553981 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:23.959572 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:23.959537 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:24.657602 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:24.657565 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:25.516460 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:25.516426 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:26.414315 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:26.414275 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:26.855203 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:46:26.855169 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:46:27.255193 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:27.255109 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:27.958207 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:27.958171 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:28.246885 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:28.246806 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:28.738658 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:28.738619 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:28.927842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:28.927802 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:29.559230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:29.559191 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:30.350927 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:30.350868 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:30.538533 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:30.538497 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:31.306929 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:31.306874 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:32.075656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:32.075618 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:32.269037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:32.269003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:32.468866 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:32.468783 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:32.953454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:32.953415 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:33.177830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:33.177754 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:46:33.528189 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:33.528145 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:33.716425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:33.716386 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:33.954609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:33.954573 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:34.657602 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:34.657567 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:35.485608 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:35.485571 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:36.089662 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.089626 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps"] Nov 02 09:46:36.094207 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.094187 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.099750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.099724 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-301be8-serving-cert\"" Nov 02 09:46:36.108536 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.108512 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps"] Nov 02 09:46:36.158244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.158212 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-maas-api-tls\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.158244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.158246 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ad92b-69c8-4e04-9203-a89302aa146e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.158446 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.158377 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-metrics-certs\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.158446 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.158411 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99rpg\" (UniqueName: \"kubernetes.io/projected/2f3ad92b-69c8-4e04-9203-a89302aa146e-kube-api-access-99rpg\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.237459 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.237391 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:36.259705 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.259665 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-maas-api-tls\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.259942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.259724 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ad92b-69c8-4e04-9203-a89302aa146e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.259942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.259826 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-metrics-certs\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.259942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.259863 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99rpg\" (UniqueName: \"kubernetes.io/projected/2f3ad92b-69c8-4e04-9203-a89302aa146e-kube-api-access-99rpg\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.260526 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.260501 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ad92b-69c8-4e04-9203-a89302aa146e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.263338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.263316 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-maas-api-tls\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.263457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.263380 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-metrics-certs\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.271988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.271956 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99rpg\" (UniqueName: \"kubernetes.io/projected/2f3ad92b-69c8-4e04-9203-a89302aa146e-kube-api-access-99rpg\") pod \"maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.418952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.418828 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:36.421216 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.421185 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:36.714148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.714115 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps"] Nov 02 09:46:36.714919 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:46:36.714848 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3ad92b_69c8_4e04_9203_a89302aa146e.slice/crio-858a858225bd50eff46d30e80da9738ae4ba9534ea0f36bcd948a1d5d1bcf148 WatchSource:0}: Error finding container 858a858225bd50eff46d30e80da9738ae4ba9534ea0f36bcd948a1d5d1bcf148: Status 404 returned error can't find the container with id 858a858225bd50eff46d30e80da9738ae4ba9534ea0f36bcd948a1d5d1bcf148 Nov 02 09:46:36.716581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.716561 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 09:46:36.807940 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.807878 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr"] Nov 02 09:46:36.808164 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.808134 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" podUID="02c00d49-a410-4910-a8ac-e7c489ce1ef8" containerName="maas-api" containerID="cri-o://194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa" gracePeriod=30 Nov 02 09:46:36.913489 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.913440 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" event={"ID":"2f3ad92b-69c8-4e04-9203-a89302aa146e","Type":"ContainerStarted","Data":"858a858225bd50eff46d30e80da9738ae4ba9534ea0f36bcd948a1d5d1bcf148"} Nov 02 09:46:36.943741 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:36.943698 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:37.049616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.049592 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:46:37.067424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.067397 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-metrics-certs\") pod \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " Nov 02 09:46:37.067579 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.067448 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c00d49-a410-4910-a8ac-e7c489ce1ef8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c00d49-a410-4910-a8ac-e7c489ce1ef8-openshift-service-ca\") pod \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " Nov 02 09:46:37.067579 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.067537 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-maas-api-tls\") pod \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " Nov 02 09:46:37.067706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.067583 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c00d49-a410-4910-a8ac-e7c489ce1ef8-kube-api-access-2m8l8\" (UniqueName: \"kubernetes.io/projected/02c00d49-a410-4910-a8ac-e7c489ce1ef8-kube-api-access-2m8l8\") pod \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\" (UID: \"02c00d49-a410-4910-a8ac-e7c489ce1ef8\") " Nov 02 09:46:37.067841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.067818 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c00d49-a410-4910-a8ac-e7c489ce1ef8-openshift-service-ca" pod "02c00d49-a410-4910-a8ac-e7c489ce1ef8" (UID: "02c00d49-a410-4910-a8ac-e7c489ce1ef8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:46:37.068016 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.067990 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c00d49-a410-4910-a8ac-e7c489ce1ef8-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:37.069756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.069729 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-metrics-certs" pod "02c00d49-a410-4910-a8ac-e7c489ce1ef8" (UID: "02c00d49-a410-4910-a8ac-e7c489ce1ef8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:46:37.069843 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.069750 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-maas-api-tls" pod "02c00d49-a410-4910-a8ac-e7c489ce1ef8" (UID: "02c00d49-a410-4910-a8ac-e7c489ce1ef8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:46:37.069843 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.069808 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c00d49-a410-4910-a8ac-e7c489ce1ef8-kube-api-access-2m8l8" pod "02c00d49-a410-4910-a8ac-e7c489ce1ef8" (UID: "02c00d49-a410-4910-a8ac-e7c489ce1ef8"). InnerVolumeSpecName "kube-api-access-2m8l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:46:37.168798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.168747 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:37.168798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.168792 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c00d49-a410-4910-a8ac-e7c489ce1ef8-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:37.169402 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.168809 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m8l8\" (UniqueName: \"kubernetes.io/projected/02c00d49-a410-4910-a8ac-e7c489ce1ef8-kube-api-access-2m8l8\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:37.242987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.242869 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:37.524378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.521430 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc"] Nov 02 09:46:37.524378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.522111 4467 cpu_manager.go:414] "containerMap: removing container" podUID="02c00d49-a410-4910-a8ac-e7c489ce1ef8" containerName="maas-api" Nov 02 09:46:37.524378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.522130 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c00d49-a410-4910-a8ac-e7c489ce1ef8" containerName="maas-api" Nov 02 09:46:37.524378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.522257 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c00d49-a410-4910-a8ac-e7c489ce1ef8" containerName="maas-api" Nov 02 09:46:37.525552 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.525529 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.537253 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.537224 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc"] Nov 02 09:46:37.572820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.572782 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.573040 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.572850 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.573040 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.572958 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.573040 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.573018 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1afc788-08f2-4260-af9e-5b3a537133ef-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.573233 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.573126 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.573233 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.573164 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.573345 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.573254 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.573345 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.573331 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzkx9\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-kube-api-access-hzkx9\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.573448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.573381 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674418 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674378 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzkx9\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-kube-api-access-hzkx9\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674436 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674480 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674521 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674557 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674602 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1afc788-08f2-4260-af9e-5b3a537133ef-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674652 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674687 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.674841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.674740 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.675124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.675024 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.675124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.675085 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.675236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.675145 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.675236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.675175 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.675535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.675515 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1afc788-08f2-4260-af9e-5b3a537133ef-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.677269 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.677244 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.677485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.677468 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.693919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.693880 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.694200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.694182 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzkx9\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-kube-api-access-hzkx9\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.782815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.782702 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:37.845341 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.845310 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:37.922257 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.922228 4467 generic.go:356] "Generic (PLEG): container finished" podID="02c00d49-a410-4910-a8ac-e7c489ce1ef8" containerID="194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa" exitCode=0 Nov 02 09:46:37.922672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.922281 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" Nov 02 09:46:37.922672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.922306 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" event={"ID":"02c00d49-a410-4910-a8ac-e7c489ce1ef8","Type":"ContainerDied","Data":"194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa"} Nov 02 09:46:37.922672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.922358 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr" event={"ID":"02c00d49-a410-4910-a8ac-e7c489ce1ef8","Type":"ContainerDied","Data":"029cfa185f566c8e2db9f655a07df9797a69e75214b3c6f6c7899d53ec157e17"} Nov 02 09:46:37.922672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.922385 4467 scope.go:122] "RemoveContainer" containerID="194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa" Nov 02 09:46:37.926770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.926740 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" event={"ID":"2f3ad92b-69c8-4e04-9203-a89302aa146e","Type":"ContainerStarted","Data":"0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d"} Nov 02 09:46:37.927051 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.927027 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:37.933079 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.933056 4467 scope.go:122] "RemoveContainer" containerID="194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa" Nov 02 09:46:37.933426 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:46:37.933403 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa\": container with ID starting with 194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa not found: ID does not exist" containerID="194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa" Nov 02 09:46:37.933526 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.933438 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa"} err="failed to get container status \"194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa\": rpc error: code = NotFound desc = could not find container \"194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa\": container with ID starting with 194ef736e6cfc8625200e4ab0fc8eda2dc8ff6b8f4258ac6e3b2414a65e683fa not found: ID does not exist" Nov 02 09:46:37.951507 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.951441 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" podStartSLOduration=1.3201594380000001 podStartE2EDuration="1.951423441s" totalImagesPullingTime="631.264003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:37.948524848 +0000 UTC m=+676.788466843" watchObservedRunningTime="2026-11-02 09:46:37.951423441 +0000 UTC m=+676.791365437" Nov 02 09:46:37.965107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.965075 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:37.965882 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.965859 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr"] Nov 02 09:46:37.970558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:37.970534 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-000ffb-7c779f9d-jlwtr"] Nov 02 09:46:38.138290 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.138259 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc"] Nov 02 09:46:38.139415 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:46:38.139391 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1afc788_08f2_4260_af9e_5b3a537133ef.slice/crio-485e0b2d610d395a355f05417883200ead0d997408ec44a2f5e66f1e4fb8cf6d WatchSource:0}: Error finding container 485e0b2d610d395a355f05417883200ead0d997408ec44a2f5e66f1e4fb8cf6d: Status 404 returned error can't find the container with id 485e0b2d610d395a355f05417883200ead0d997408ec44a2f5e66f1e4fb8cf6d Nov 02 09:46:38.141697 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.141660 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:38.141781 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.141725 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:38.141781 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.141756 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:38.160107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.160079 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:38.353578 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.353526 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:38.642921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.628857 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6"] Nov 02 09:46:38.642921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.629128 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" containerID="cri-o://54b76473dec66b8f44d0cfb46747ee011ab935cd829a635a5905a59d29caf97a" gracePeriod=30 Nov 02 09:46:38.739596 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.739554 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:38.934864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.934778 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" event={"ID":"b1afc788-08f2-4260-af9e-5b3a537133ef","Type":"ContainerStarted","Data":"d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d"} Nov 02 09:46:38.934864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:38.934836 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" event={"ID":"b1afc788-08f2-4260-af9e-5b3a537133ef","Type":"ContainerStarted","Data":"485e0b2d610d395a355f05417883200ead0d997408ec44a2f5e66f1e4fb8cf6d"} Nov 02 09:46:39.224451 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:39.224319 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:39.331433 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:39.331401 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c00d49-a410-4910-a8ac-e7c489ce1ef8" path="/var/lib/kubelet/pods/02c00d49-a410-4910-a8ac-e7c489ce1ef8/volumes" Nov 02 09:46:39.846118 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:39.846075 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:39.847825 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:39.847800 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 09:46:39.847920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:39.847851 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 09:46:39.979511 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:39.979470 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:40.776503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:40.776465 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:40.845979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:40.845940 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 09:46:40.846162 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:40.846008 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 09:46:41.578139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:41.578101 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:41.846432 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:41.846344 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 09:46:41.846432 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:41.846417 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 09:46:42.392292 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:42.392215 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:42.846581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:42.846543 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 09:46:42.847046 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:42.846613 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 09:46:43.106913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:43.106768 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:43.845794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:43.845756 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 09:46:43.846010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:43.845820 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 09:46:43.953691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:43.953189 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:46:43.981340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:43.980512 4467 generic.go:356] "Generic (PLEG): container finished" podID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerID="54b76473dec66b8f44d0cfb46747ee011ab935cd829a635a5905a59d29caf97a" exitCode=0 Nov 02 09:46:43.981340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:43.980683 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" event={"ID":"76dea2d8-48a7-4f49-be4c-82d2f607554d","Type":"ContainerDied","Data":"54b76473dec66b8f44d0cfb46747ee011ab935cd829a635a5905a59d29caf97a"} Nov 02 09:46:43.981340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:43.980744 4467 scope.go:122] "RemoveContainer" containerID="f2ee28c74714430b486edf1a33eba85556d9fa8e6f11b11f47510d8309fd2859" Nov 02 09:46:44.018009 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.012403 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podStartSLOduration=7.012380067 podStartE2EDuration="7.012380067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:39.009419976 +0000 UTC m=+677.849361971" watchObservedRunningTime="2026-11-02 09:46:44.012380067 +0000 UTC m=+682.852322063" Nov 02 09:46:44.124428 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.124190 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:46:44.182642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.182601 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76dea2d8-48a7-4f49-be4c-82d2f607554d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dea2d8-48a7-4f49-be4c-82d2f607554d-istiod-ca-cert\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.183487 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.183324 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dea2d8-48a7-4f49-be4c-82d2f607554d-istiod-ca-cert" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:46:44.184107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.184085 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-podinfo\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.184298 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.184281 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-kube-api-access-r55w9\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-kube-api-access-r55w9\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.184767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.184749 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-envoy\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.185261 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.185245 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-data\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.185838 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.185695 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-data" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:46:44.186382 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.186331 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-certs" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.186056 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-certs\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.186579 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-token\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-token\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.186616 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-credential-socket\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.187478 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-socket" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.187539 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-credential-socket" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.187578 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-socket\") pod \"76dea2d8-48a7-4f49-be4c-82d2f607554d\" (UID: \"76dea2d8-48a7-4f49-be4c-82d2f607554d\") " Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.188059 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.188079 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.188095 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.188111 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.188148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.188125 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dea2d8-48a7-4f49-be4c-82d2f607554d-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.191118 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.191082 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-token" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:46:44.191542 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.191519 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-envoy" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:46:44.192274 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.192229 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-podinfo" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:46:44.193600 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.193565 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-kube-api-access-r55w9" pod "76dea2d8-48a7-4f49-be4c-82d2f607554d" (UID: "76dea2d8-48a7-4f49-be4c-82d2f607554d"). InnerVolumeSpecName "kube-api-access-r55w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:46:44.269953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.269782 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:44.289498 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.289448 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.289498 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.289491 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r55w9\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-kube-api-access-r55w9\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.289770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.289512 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.289770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.289528 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dea2d8-48a7-4f49-be4c-82d2f607554d-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:44.747361 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.747317 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:44.849377 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.849334 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 02 09:46:44.849678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.849645 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 02 09:46:45.002982 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.995982 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" event={"ID":"76dea2d8-48a7-4f49-be4c-82d2f607554d","Type":"ContainerDied","Data":"a7f2f79eb0ec53d5adf4dfebf5c17c1486044a95c066aa39c721448ebaa07d23"} Nov 02 09:46:45.003657 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:45.003625 4467 scope.go:122] "RemoveContainer" containerID="54b76473dec66b8f44d0cfb46747ee011ab935cd829a635a5905a59d29caf97a" Nov 02 09:46:45.003982 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:44.996332 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6" Nov 02 09:46:45.103987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:45.103953 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6"] Nov 02 09:46:45.125928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:45.117854 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-9xtr6"] Nov 02 09:46:45.155429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:45.155393 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:45.345686 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:45.345646 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" path="/var/lib/kubelet/pods/76dea2d8-48a7-4f49-be4c-82d2f607554d/volumes" Nov 02 09:46:45.868859 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:45.868828 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:46.667710 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:46.667660 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:46.820711 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:46.819678 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:46.820711 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:46.820491 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:46.824166 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:46.824139 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:46:47.464853 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:47.464814 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:48.164603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:48.164572 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:49.101090 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:49.101059 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:49.880291 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:49.880254 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:50.631222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:50.631185 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:50.868714 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:50.868671 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:51.063171 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:51.063131 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:51.448516 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:51.448423 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:51.678725 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:51.678691 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:52.133148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.133116 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2"] Nov 02 09:46:52.133663 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.133647 4467 cpu_manager.go:414] "containerMap: removing container" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" Nov 02 09:46:52.133706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.133664 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" Nov 02 09:46:52.133706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.133683 4467 cpu_manager.go:414] "containerMap: removing container" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" Nov 02 09:46:52.133706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.133688 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" Nov 02 09:46:52.133804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.133775 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" Nov 02 09:46:52.133804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.133784 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dea2d8-48a7-4f49-be4c-82d2f607554d" containerName="istio-proxy" Nov 02 09:46:52.139340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.139312 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.149497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.149462 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-bde944-openshift-default-dockercfg-hb9wq\"" Nov 02 09:46:52.150601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.150572 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2"] Nov 02 09:46:52.166930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.166880 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:52.257312 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257272 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.257514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257325 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.257514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257426 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.257659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257530 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918367-c36f-4816-8674-c33e3beb85ce-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.257659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257605 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c867x\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-kube-api-access-c867x\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.257659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257647 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.257776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257706 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.257776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257743 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.257851 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.257812 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918367-c36f-4816-8674-c33e3beb85ce-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.358565 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358523 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918367-c36f-4816-8674-c33e3beb85ce-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.358768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358578 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c867x\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-kube-api-access-c867x\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.358768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358609 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.358768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358675 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.358768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358724 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358771 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918367-c36f-4816-8674-c33e3beb85ce-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358821 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358865 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.358960 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.359106 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359441 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.359413 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359556 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.359448 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359556 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.359515 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918367-c36f-4816-8674-c33e3beb85ce-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.359649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.359557 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.361020 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.360993 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.361148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.361127 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918367-c36f-4816-8674-c33e3beb85ce-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.371928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.371876 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.371928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.371869 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c867x\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-kube-api-access-c867x\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.372945 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.372919 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:52.456165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.456070 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:52.773584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.773558 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2"] Nov 02 09:46:52.775460 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:46:52.775429 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a918367_c36f_4816_8674_c33e3beb85ce.slice/crio-4de6eeb2122f27ffa89598decfa2f7b5e2fdbb96815f5a4cc219b9d36cd970ea WatchSource:0}: Error finding container 4de6eeb2122f27ffa89598decfa2f7b5e2fdbb96815f5a4cc219b9d36cd970ea: Status 404 returned error can't find the container with id 4de6eeb2122f27ffa89598decfa2f7b5e2fdbb96815f5a4cc219b9d36cd970ea Nov 02 09:46:52.777688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.777655 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:52.777790 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.777723 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:52.777790 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.777752 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:46:52.813071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.813035 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:52.957318 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:52.957283 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:46:53.001367 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.001323 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:53.048068 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.047987 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" event={"ID":"1a918367-c36f-4816-8674-c33e3beb85ce","Type":"ContainerStarted","Data":"d90621427d7a6d8725808b02edcf5504cd273a087bcddcc032f7f64cd1d7539b"} Nov 02 09:46:53.048068 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.048054 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" event={"ID":"1a918367-c36f-4816-8674-c33e3beb85ce","Type":"ContainerStarted","Data":"4de6eeb2122f27ffa89598decfa2f7b5e2fdbb96815f5a4cc219b9d36cd970ea"} Nov 02 09:46:53.083065 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.083003 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" podStartSLOduration=1.082983198 podStartE2EDuration="1.082983198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:53.077090422 +0000 UTC m=+691.917032418" watchObservedRunningTime="2026-11-02 09:46:53.082983198 +0000 UTC m=+691.922925194" Nov 02 09:46:53.293960 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.293869 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:53.456258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.456226 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:53.457582 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.457556 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 09:46:53.457709 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.457603 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 09:46:53.472128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:53.472098 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:54.047439 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:54.047397 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:54.457059 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:54.457014 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 09:46:54.457591 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:54.457074 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 09:46:55.149755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:55.149717 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:55.457190 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:55.457098 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 09:46:55.457190 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:55.457160 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 09:46:56.304919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:56.301682 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:56.456873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:56.456842 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 09:46:56.457087 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:56.456966 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 09:46:56.517938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:56.517748 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:56.701113 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:56.701079 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:56.856287 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:46:56.856248 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:46:57.022592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.022269 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs"] Nov 02 09:46:57.022592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.022450 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" podUID="929261f1-a83d-4e36-98f8-f9f4cec09ef9" containerName="maas-api" containerID="cri-o://1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4" gracePeriod=30 Nov 02 09:46:57.163627 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.163554 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:57.367375 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.367344 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:46:57.456609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.456573 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 09:46:57.456761 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.456632 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 09:46:57.560406 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.560365 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-metrics-certs\") pod \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " Nov 02 09:46:57.560953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.560505 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929261f1-a83d-4e36-98f8-f9f4cec09ef9-kube-api-access-xsx4b\" (UniqueName: \"kubernetes.io/projected/929261f1-a83d-4e36-98f8-f9f4cec09ef9-kube-api-access-xsx4b\") pod \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " Nov 02 09:46:57.560953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.560559 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929261f1-a83d-4e36-98f8-f9f4cec09ef9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929261f1-a83d-4e36-98f8-f9f4cec09ef9-openshift-service-ca\") pod \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " Nov 02 09:46:57.560953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.560581 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-maas-api-tls\") pod \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\" (UID: \"929261f1-a83d-4e36-98f8-f9f4cec09ef9\") " Nov 02 09:46:57.561274 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.561249 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929261f1-a83d-4e36-98f8-f9f4cec09ef9-openshift-service-ca" pod "929261f1-a83d-4e36-98f8-f9f4cec09ef9" (UID: "929261f1-a83d-4e36-98f8-f9f4cec09ef9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:46:57.562801 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.562762 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-metrics-certs" pod "929261f1-a83d-4e36-98f8-f9f4cec09ef9" (UID: "929261f1-a83d-4e36-98f8-f9f4cec09ef9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:46:57.563135 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.563097 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-maas-api-tls" pod "929261f1-a83d-4e36-98f8-f9f4cec09ef9" (UID: "929261f1-a83d-4e36-98f8-f9f4cec09ef9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:46:57.563376 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.563350 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929261f1-a83d-4e36-98f8-f9f4cec09ef9-kube-api-access-xsx4b" pod "929261f1-a83d-4e36-98f8-f9f4cec09ef9" (UID: "929261f1-a83d-4e36-98f8-f9f4cec09ef9"). InnerVolumeSpecName "kube-api-access-xsx4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:46:57.664830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.664678 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:57.664830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.664718 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsx4b\" (UniqueName: \"kubernetes.io/projected/929261f1-a83d-4e36-98f8-f9f4cec09ef9-kube-api-access-xsx4b\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:57.664830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.664733 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929261f1-a83d-4e36-98f8-f9f4cec09ef9-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:57.664830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.664746 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929261f1-a83d-4e36-98f8-f9f4cec09ef9-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:46:57.721660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.721245 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:57.939971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:57.923446 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:58.116541 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.116496 4467 generic.go:356] "Generic (PLEG): container finished" podID="929261f1-a83d-4e36-98f8-f9f4cec09ef9" containerID="1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4" exitCode=0 Nov 02 09:46:58.116541 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.116543 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" Nov 02 09:46:58.116754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.116578 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" event={"ID":"929261f1-a83d-4e36-98f8-f9f4cec09ef9","Type":"ContainerDied","Data":"1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4"} Nov 02 09:46:58.116754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.116633 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs" event={"ID":"929261f1-a83d-4e36-98f8-f9f4cec09ef9","Type":"ContainerDied","Data":"c672765614bfbcd86fa03457b1e67de76c298b4e3de1b6d5587c4a2c5d191a30"} Nov 02 09:46:58.116754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.116662 4467 scope.go:122] "RemoveContainer" containerID="1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4" Nov 02 09:46:58.127866 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.127840 4467 scope.go:122] "RemoveContainer" containerID="1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4" Nov 02 09:46:58.128216 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:46:58.128166 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4\": container with ID starting with 1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4 not found: ID does not exist" containerID="1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4" Nov 02 09:46:58.128216 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.128191 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4"} err="failed to get container status \"1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4\": rpc error: code = NotFound desc = could not find container \"1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4\": container with ID starting with 1895e5d2f482bb46825fc23d63bb8ec918796aac8a66e260bcbba51bf46838b4 not found: ID does not exist" Nov 02 09:46:58.140756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.140724 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs"] Nov 02 09:46:58.144589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.144560 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d2e277-58cb59c564-gfgqs"] Nov 02 09:46:58.456753 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.456723 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 09:46:58.457047 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.457021 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 09:46:58.699817 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.698367 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:58.895471 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:58.895439 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:59.210033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:59.209934 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:46:59.334247 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:59.334211 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929261f1-a83d-4e36-98f8-f9f4cec09ef9" path="/var/lib/kubelet/pods/929261f1-a83d-4e36-98f8-f9f4cec09ef9/volumes" Nov 02 09:46:59.459776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:59.457272 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm"] Nov 02 09:46:59.459776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:59.457526 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" containerID="cri-o://0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4" gracePeriod=30 Nov 02 09:46:59.530651 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:59.530618 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:59.531085 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:59.531062 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:59.531944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:59.531883 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:46:59.580916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:46:59.579171 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:00.415932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:00.415886 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:01.382914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.382849 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:01.609046 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.609010 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:01.927818 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.927776 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb"] Nov 02 09:47:01.928654 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.928631 4467 cpu_manager.go:414] "containerMap: removing container" podUID="929261f1-a83d-4e36-98f8-f9f4cec09ef9" containerName="maas-api" Nov 02 09:47:01.928738 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.928658 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929261f1-a83d-4e36-98f8-f9f4cec09ef9" containerName="maas-api" Nov 02 09:47:01.928828 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.928812 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="929261f1-a83d-4e36-98f8-f9f4cec09ef9" containerName="maas-api" Nov 02 09:47:01.934274 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.934249 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:01.946981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.946945 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb"] Nov 02 09:47:01.947301 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:01.947281 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-629129-openshift-default-dockercfg-mggz7\"" Nov 02 09:47:02.015133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015084 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.015335 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015145 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.015335 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015278 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjpr\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-kube-api-access-ckjpr\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.015335 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015326 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.015496 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015402 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.015496 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015439 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea47c669-9830-478b-8aa5-0e140e5660f6-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.015496 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015479 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.015657 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015519 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.015657 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.015556 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.061097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.061061 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:02.116592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116550 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.116592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116598 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.116837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116645 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.116837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116722 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.116837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116759 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.116837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116809 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjpr\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-kube-api-access-ckjpr\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.116837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116835 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.117134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116925 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.117134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.116967 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea47c669-9830-478b-8aa5-0e140e5660f6-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.117134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.117036 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.117134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.117088 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.117342 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.117211 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.117413 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.117389 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.117952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.117930 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea47c669-9830-478b-8aa5-0e140e5660f6-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.119403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.119382 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.120217 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.120199 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.129798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.129774 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjpr\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-kube-api-access-ckjpr\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.130152 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.130133 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.246098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.245965 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:02.257638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.257605 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:02.490672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.490636 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:02.549792 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.549764 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb"] Nov 02 09:47:02.551539 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:47:02.551511 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea47c669_9830_478b_8aa5_0e140e5660f6.slice/crio-8a1d484d10756af89c443b2618a3323f5872b7a660f5a35a56b59a602e99f8d4 WatchSource:0}: Error finding container 8a1d484d10756af89c443b2618a3323f5872b7a660f5a35a56b59a602e99f8d4: Status 404 returned error can't find the container with id 8a1d484d10756af89c443b2618a3323f5872b7a660f5a35a56b59a602e99f8d4 Nov 02 09:47:02.554179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.554147 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:02.554261 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.554211 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:02.554261 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:02.554241 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:03.150519 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.150486 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" event={"ID":"ea47c669-9830-478b-8aa5-0e140e5660f6","Type":"ContainerStarted","Data":"5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063"} Nov 02 09:47:03.150972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.150545 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" event={"ID":"ea47c669-9830-478b-8aa5-0e140e5660f6","Type":"ContainerStarted","Data":"8a1d484d10756af89c443b2618a3323f5872b7a660f5a35a56b59a602e99f8d4"} Nov 02 09:47:03.174878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.174824 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podStartSLOduration=2.174810789 podStartE2EDuration="2.174810789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:03.172992243 +0000 UTC m=+702.012934233" watchObservedRunningTime="2026-11-02 09:47:03.174810789 +0000 UTC m=+702.014752784" Nov 02 09:47:03.258061 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.258021 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:03.263416 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.263385 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:03.300504 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.300463 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58"] Nov 02 09:47:03.317949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.310933 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58"] Nov 02 09:47:03.317949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.311707 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.317949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.316417 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8743a5-serving-cert\"" Nov 02 09:47:03.344142 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.344106 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.344343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.344200 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-metrics-certs\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.344343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.344281 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5pk7\" (UniqueName: \"kubernetes.io/projected/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-kube-api-access-z5pk7\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.344343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.344317 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.445478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.445378 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.445478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.445473 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-metrics-certs\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.445725 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:47:03.445554 4467 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-serving-cert: secret "maas-api-e2e-inf-b-8743a5-serving-cert" not found Nov 02 09:47:03.445725 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:47:03.445636 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls podName:6903cdc1-3ed6-4ff1-9225-096d1d7e65a9 nodeName:}" failed. No retries permitted until 2026-11-02 09:47:03.945614773 +0000 UTC m=+702.785556748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls") pod "maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" (UID: "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9") : secret "maas-api-e2e-inf-b-8743a5-serving-cert" not found Nov 02 09:47:03.445855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.445556 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5pk7\" (UniqueName: \"kubernetes.io/projected/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-kube-api-access-z5pk7\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.445855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.445828 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.446781 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.446754 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.448684 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.448661 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-metrics-certs\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.457710 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.457592 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:03.461644 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.461618 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5pk7\" (UniqueName: \"kubernetes.io/projected/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-kube-api-access-z5pk7\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.480117 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.480074 4467 patch_prober.go:29] interesting pod/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 09:47:03.480276 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.480134 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 09:47:03.658399 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.658366 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:03.953615 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.953574 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.956400 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.956363 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:03.965820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:03.965788 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:04.156603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.156567 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:04.157665 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.157646 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:04.315732 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.315702 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58"] Nov 02 09:47:04.317259 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:47:04.317224 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6903cdc1_3ed6_4ff1_9225_096d1d7e65a9.slice/crio-8c364412ef678d40397cf6927f82a19fb0ae93338fc5c6fd26bf9bfe727f2f08 WatchSource:0}: Error finding container 8c364412ef678d40397cf6927f82a19fb0ae93338fc5c6fd26bf9bfe727f2f08: Status 404 returned error can't find the container with id 8c364412ef678d40397cf6927f82a19fb0ae93338fc5c6fd26bf9bfe727f2f08 Nov 02 09:47:04.340912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.340857 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:04.726506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.726479 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:47:04.763597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.763566 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2333ec-e848-48dd-a738-039a7dd0a74f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2333ec-e848-48dd-a738-039a7dd0a74f-istiod-ca-cert\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.763758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.763613 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-certs\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.763758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.763658 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-credential-socket\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.763758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.763689 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-socket\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.763758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.763731 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-kube-api-access-xwcvp\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-kube-api-access-xwcvp\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.764043 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.763779 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-envoy\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.764043 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.763965 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-podinfo\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.764043 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.763995 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-data\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.764196 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.764045 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-token\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-token\") pod \"cc2333ec-e848-48dd-a738-039a7dd0a74f\" (UID: \"cc2333ec-e848-48dd-a738-039a7dd0a74f\") " Nov 02 09:47:04.764196 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.764100 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-credential-socket" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:47:04.764196 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.764016 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2333ec-e848-48dd-a738-039a7dd0a74f-istiod-ca-cert" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:47:04.764376 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.764343 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-data" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:47:04.764728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.764567 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:04.764728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.764591 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2333ec-e848-48dd-a738-039a7dd0a74f-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:04.764728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.764608 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:04.764963 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.764853 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-certs" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:47:04.765143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.765122 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-socket" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:47:04.766930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.766870 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-token" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:47:04.766930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.766876 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-kube-api-access-xwcvp" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "kube-api-access-xwcvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:47:04.767212 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.767187 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-podinfo" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:47:04.767326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.767194 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-envoy" pod "cc2333ec-e848-48dd-a738-039a7dd0a74f" (UID: "cc2333ec-e848-48dd-a738-039a7dd0a74f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:47:04.866279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.866239 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:04.866279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.866281 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:04.866470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.866300 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwcvp\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-kube-api-access-xwcvp\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:04.866470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.866316 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:04.866470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.866331 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:04.866470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:04.866347 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2333ec-e848-48dd-a738-039a7dd0a74f-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:47:05.149794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.149758 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:05.162983 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.162935 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" event={"ID":"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9","Type":"ContainerStarted","Data":"c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf"} Nov 02 09:47:05.163471 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.162997 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" event={"ID":"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9","Type":"ContainerStarted","Data":"8c364412ef678d40397cf6927f82a19fb0ae93338fc5c6fd26bf9bfe727f2f08"} Nov 02 09:47:05.163471 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.163103 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:05.164807 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.164782 4467 generic.go:356] "Generic (PLEG): container finished" podID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerID="0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4" exitCode=0 Nov 02 09:47:05.164948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.164823 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" Nov 02 09:47:05.164948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.164871 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" event={"ID":"cc2333ec-e848-48dd-a738-039a7dd0a74f","Type":"ContainerDied","Data":"0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4"} Nov 02 09:47:05.165032 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.164955 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm" event={"ID":"cc2333ec-e848-48dd-a738-039a7dd0a74f","Type":"ContainerDied","Data":"4741f17b7bb7e065dd0f08b6ebdac929052085e8ea4bd52e34a0cd814a82ada2"} Nov 02 09:47:05.165032 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.164986 4467 scope.go:122] "RemoveContainer" containerID="0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4" Nov 02 09:47:05.174518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.174495 4467 scope.go:122] "RemoveContainer" containerID="0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e" Nov 02 09:47:05.183728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.183674 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" podStartSLOduration=1.718149511 podStartE2EDuration="2.183656929s" totalImagesPullingTime="465.507418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:05.183138936 +0000 UTC m=+704.023080931" watchObservedRunningTime="2026-11-02 09:47:05.183656929 +0000 UTC m=+704.023598926" Nov 02 09:47:05.184039 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.184020 4467 scope.go:122] "RemoveContainer" containerID="0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4" Nov 02 09:47:05.184334 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:47:05.184306 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4\": container with ID starting with 0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4 not found: ID does not exist" containerID="0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4" Nov 02 09:47:05.184418 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.184344 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4"} err="failed to get container status \"0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4\": rpc error: code = NotFound desc = could not find container \"0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4\": container with ID starting with 0d8899e7762e8ec28c247c3efebbe047d2e48922fbe26ecc79b6f30bae26a0e4 not found: ID does not exist" Nov 02 09:47:05.184418 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.184367 4467 scope.go:122] "RemoveContainer" containerID="0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e" Nov 02 09:47:05.184645 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:47:05.184621 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e\": container with ID starting with 0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e not found: ID does not exist" containerID="0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e" Nov 02 09:47:05.184705 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.184652 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e"} err="failed to get container status \"0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e\": rpc error: code = NotFound desc = could not find container \"0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e\": container with ID starting with 0ea9f2a37ed354640a99e798e750519ed401a6f1b68569aed16f185ff1576f8e not found: ID does not exist" Nov 02 09:47:05.198865 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.198839 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm"] Nov 02 09:47:05.203490 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.203465 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d2e277-openshift-default-94fd97fdf-t7vdm"] Nov 02 09:47:05.333288 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.333250 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" path="/var/lib/kubelet/pods/cc2333ec-e848-48dd-a738-039a7dd0a74f/volumes" Nov 02 09:47:05.648546 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.648503 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:05.830684 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:05.830644 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:06.058116 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:06.058084 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:06.847617 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:06.847581 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:07.029621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.029583 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:07.463288 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.463246 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t"] Nov 02 09:47:07.463788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.463773 4467 cpu_manager.go:414] "containerMap: removing container" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" Nov 02 09:47:07.463788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.463786 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" Nov 02 09:47:07.463959 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.463866 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" Nov 02 09:47:07.463959 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.463874 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" Nov 02 09:47:07.467425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.467404 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.482696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.482667 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t"] Nov 02 09:47:07.489339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.486399 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl"] Nov 02 09:47:07.489339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.487237 4467 cpu_manager.go:414] "containerMap: removing container" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" Nov 02 09:47:07.489339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.487256 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2333ec-e848-48dd-a738-039a7dd0a74f" containerName="istio-proxy" Nov 02 09:47:07.491284 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.491260 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.522247 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522212 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.522412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522265 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.522412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522301 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.522412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522364 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.522412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522390 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.522642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522438 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7f76\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-kube-api-access-g7f76\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.522642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522475 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.522642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522502 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.522642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.522589 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.528581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.528540 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl"] Nov 02 09:47:07.624127 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624091 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.624320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624136 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.624320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624162 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.624320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624244 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.624320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624278 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.624554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624365 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.624554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624505 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.624554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624532 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.624708 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624569 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.624708 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624601 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.624708 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624642 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkds\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-kube-api-access-8qkds\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.624860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624710 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.624860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624745 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7f76\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-kube-api-access-g7f76\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.624860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624820 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.625021 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624864 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.625021 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624942 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.625021 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.624990 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.625183 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.625024 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.625183 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.625031 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.625183 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.625093 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.625458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.625429 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.625544 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.625491 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.625722 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.625700 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.626762 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.626737 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.627153 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.627133 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.637458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.637425 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.637590 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.637506 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7f76\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-kube-api-access-g7f76\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.726584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.726481 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.726584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.726561 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.726837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.726586 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.727109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.727075 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.727219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.727144 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkds\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-kube-api-access-8qkds\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.727219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.727150 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.727459 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.727439 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.727518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.727465 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.727579 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.727558 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.727636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.727603 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.727689 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.727661 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.728529 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.728244 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.728529 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.728487 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.729755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.729731 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.730689 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.730663 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.731267 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.731239 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.751485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.751455 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.751741 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.751717 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkds\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-kube-api-access-8qkds\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:07.755708 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.755682 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:07.783436 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.783392 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:07.839798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:07.839761 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:08.077696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.077670 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t"] Nov 02 09:47:08.079020 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:47:08.078996 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ada5f7d_1d30_4bc3_b1ac_9d700d1e4426.slice/crio-602014c2680aaf401ff27662f5117ea340db7666491029d00314f87941f0d990 WatchSource:0}: Error finding container 602014c2680aaf401ff27662f5117ea340db7666491029d00314f87941f0d990: Status 404 returned error can't find the container with id 602014c2680aaf401ff27662f5117ea340db7666491029d00314f87941f0d990 Nov 02 09:47:08.081443 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.081414 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:08.081528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.081476 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:08.081528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.081503 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:08.153467 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.153438 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl"] Nov 02 09:47:08.155394 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:47:08.155370 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c5bc4b_f148_451e_b39e_bf9c717f37b2.slice/crio-2beea59ba136adac0b95b8f627a8239ee941d7b0787880c84b9ffd9ff819723c WatchSource:0}: Error finding container 2beea59ba136adac0b95b8f627a8239ee941d7b0787880c84b9ffd9ff819723c: Status 404 returned error can't find the container with id 2beea59ba136adac0b95b8f627a8239ee941d7b0787880c84b9ffd9ff819723c Nov 02 09:47:08.157946 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.157876 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:08.158021 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.157985 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:08.158061 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.158019 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:08.186011 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.185967 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" event={"ID":"07c5bc4b-f148-451e-b39e-bf9c717f37b2","Type":"ContainerStarted","Data":"2beea59ba136adac0b95b8f627a8239ee941d7b0787880c84b9ffd9ff819723c"} Nov 02 09:47:08.187820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.187793 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" event={"ID":"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426","Type":"ContainerStarted","Data":"d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73"} Nov 02 09:47:08.187948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.187839 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" event={"ID":"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426","Type":"ContainerStarted","Data":"602014c2680aaf401ff27662f5117ea340db7666491029d00314f87941f0d990"} Nov 02 09:47:08.215653 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.215586 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podStartSLOduration=1.215566372 podStartE2EDuration="1.215566372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:08.210061947 +0000 UTC m=+707.050003943" watchObservedRunningTime="2026-11-02 09:47:08.215566372 +0000 UTC m=+707.055508377" Nov 02 09:47:08.443145 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.443053 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:08.783556 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:08.783460 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:09.152969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.142486 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:09.242946 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.240951 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" event={"ID":"07c5bc4b-f148-451e-b39e-bf9c717f37b2","Type":"ContainerStarted","Data":"4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46"} Nov 02 09:47:09.277121 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.277071 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podStartSLOduration=2.277056323 podStartE2EDuration="2.277056323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:09.272371485 +0000 UTC m=+708.112313492" watchObservedRunningTime="2026-11-02 09:47:09.277056323 +0000 UTC m=+708.116998317" Nov 02 09:47:09.575871 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.575841 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:47:09.808915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.808031 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 09:47:09.808915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.808101 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 09:47:09.842977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.842508 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:09.853917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.851075 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 09:47:09.853917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.851138 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 09:47:09.932916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:09.928655 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:10.713820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:10.713775 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:10.786473 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:10.786434 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 09:47:10.786801 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:10.786748 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 09:47:10.840438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:10.840402 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 09:47:10.840755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:10.840707 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 09:47:11.140915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:11.135474 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:11.390980 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:11.390413 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:11.491916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:11.487452 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:47:11.698977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:11.698622 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:11.814699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:11.814661 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 09:47:11.852914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:11.846230 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 09:47:11.861918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:11.853912 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 09:47:11.869915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:11.865987 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 09:47:12.602919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:12.602316 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:12.789914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:12.789293 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 09:47:12.789914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:12.789364 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 09:47:12.843915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:12.843644 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 09:47:12.843915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:12.843707 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 09:47:13.461436 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:13.461401 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:13.786979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:13.786285 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 09:47:13.786979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:13.786355 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 09:47:13.846099 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:13.846063 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 09:47:13.846715 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:13.846663 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 09:47:14.170677 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:14.170638 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:14.790728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:14.790691 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 09:47:14.791076 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:14.791034 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 09:47:14.852834 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:14.850300 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 09:47:14.852834 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:14.850362 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 09:47:14.911931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:14.905383 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:15.668578 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:15.668542 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:15.784345 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:15.784301 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 09:47:15.784536 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:15.784380 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 09:47:15.840407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:15.840358 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 09:47:15.840613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:15.840420 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 09:47:16.098105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.098066 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp"] Nov 02 09:47:16.104495 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.104466 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.109942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.109618 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 09:47:16.114666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.114634 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp"] Nov 02 09:47:16.115357 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.115211 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 09:47:16.115512 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.115253 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 09:47:16.115592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.115381 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8zrbm\"" Nov 02 09:47:16.177406 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.177367 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.177597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.177448 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.177597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.177537 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.177597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.177580 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.177770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.177633 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.177770 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.177676 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887bn\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-kube-api-access-887bn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.177873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.177819 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.278817 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.278771 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.278817 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.278819 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.279082 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.278857 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.279082 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.278931 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887bn\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-kube-api-access-887bn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.279082 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.279052 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.279246 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.279108 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.279246 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.279159 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.279727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.279696 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.281573 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.281542 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.281862 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.281842 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.281969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.281868 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.282436 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.282408 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.295834 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.295030 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.298330 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.298302 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887bn\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-kube-api-access-887bn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fmbxp\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.376819 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.376736 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:16.442431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.442389 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:16.756372 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.755983 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp"] Nov 02 09:47:16.759788 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:47:16.759756 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69aa7fba_fb2f_45bd_a3ae_ef602853efdd.slice/crio-e5295b09cb94a6850c347cd5a4385c0e4d0e452d7a6c77cb1e2fdd98b5d5deb2 WatchSource:0}: Error finding container e5295b09cb94a6850c347cd5a4385c0e4d0e452d7a6c77cb1e2fdd98b5d5deb2: Status 404 returned error can't find the container with id e5295b09cb94a6850c347cd5a4385c0e4d0e452d7a6c77cb1e2fdd98b5d5deb2 Nov 02 09:47:16.784110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.784067 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 09:47:16.784296 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.784127 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 09:47:16.841081 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.841046 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 09:47:16.841252 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.841110 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 09:47:16.879574 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:16.879535 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" event={"ID":"69aa7fba-fb2f-45bd-a3ae-ef602853efdd","Type":"ContainerStarted","Data":"e5295b09cb94a6850c347cd5a4385c0e4d0e452d7a6c77cb1e2fdd98b5d5deb2"} Nov 02 09:47:17.331949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:17.331807 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:18.231033 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:18.230986 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:18.844405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:18.844182 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:47:18.844405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:18.844249 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:47:18.932932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:18.909877 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:47:18.932932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:18.909963 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:47:18.961665 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:18.961625 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:19.753953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:19.746303 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:19.892040 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:19.891999 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:47:19.892651 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:19.892620 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:47:19.938551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:19.937757 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:47:19.938551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:19.937826 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:47:20.016648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:20.016521 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:20.028437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:20.028258 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:20.030660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:20.029779 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:47:20.052273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:20.052243 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:20.052870 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:20.052794 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:20.053901 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:20.053868 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:47:20.520992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:20.520956 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:21.424346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:21.424173 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:21.869950 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:21.866683 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:22.588584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:22.588497 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:23.354926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:23.348781 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:23.716150 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:23.716043 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:24.376250 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:24.376191 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:25.266842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:25.266806 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:26.154971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:26.154207 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:26.154971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:26.154297 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:26.337847 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:26.337816 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:26.881126 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:47:26.881082 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:47:27.168458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:27.168363 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" event={"ID":"69aa7fba-fb2f-45bd-a3ae-ef602853efdd","Type":"ContainerStarted","Data":"21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4"} Nov 02 09:47:27.168638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:27.168525 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:27.170631 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:27.170605 4467 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fmbxp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 09:47:27.170785 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:27.170657 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" podUID="69aa7fba-fb2f-45bd-a3ae-ef602853efdd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 09:47:27.209927 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:27.209839 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" podStartSLOduration=1.81942154 podStartE2EDuration="11.209816718s" totalImagesPullingTime="9.390395178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:27.203079369 +0000 UTC m=+726.043021365" watchObservedRunningTime="2026-11-02 09:47:27.209816718 +0000 UTC m=+726.049758709" Nov 02 09:47:27.318593 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:27.318556 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:28.173070 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:28.173028 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:28.176578 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:28.176553 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 09:47:29.140755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:29.140682 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:29.352937 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:29.352862 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:30.258271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:30.258233 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:30.518647 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:30.518548 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:30.967145 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:30.967115 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:31.272238 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:31.272111 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:32.074227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.074191 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:32.208388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.208351 4467 generic.go:356] "Generic (PLEG): container finished" podID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerID="5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063" exitCode=0 Nov 02 09:47:32.208560 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.208431 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" event={"ID":"ea47c669-9830-478b-8aa5-0e140e5660f6","Type":"ContainerDied","Data":"5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063"} Nov 02 09:47:32.208856 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.208840 4467 scope.go:122] "RemoveContainer" containerID="5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063" Nov 02 09:47:32.210366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.210340 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:32.210471 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.210405 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:32.210471 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.210449 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:32.384157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.384048 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:32.596466 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.596426 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:47:32.923423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:32.923371 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:33.114278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:33.114240 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:33.236990 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:33.236848 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" event={"ID":"ea47c669-9830-478b-8aa5-0e140e5660f6","Type":"ContainerStarted","Data":"11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248"} Nov 02 09:47:33.258741 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:33.258704 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:33.260046 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:33.260021 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 09:47:33.260266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:33.260240 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 09:47:33.461856 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:33.461824 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:34.157526 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:34.157489 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:34.258926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:34.258858 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 09:47:34.259141 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:34.258972 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 09:47:34.788557 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:34.788473 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:35.274424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:35.274386 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 09:47:35.350941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:35.305339 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 09:47:35.859666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:35.859626 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:36.115334 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:36.115200 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:47:36.138044 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:36.119002 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:47:36.138454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:36.138396 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:47:36.282061 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:36.274044 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 09:47:36.282061 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:36.274125 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 09:47:36.307067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:36.307015 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:47:36.549980 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:36.549513 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:37.258153 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:37.258108 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 09:47:37.258365 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:37.258174 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 09:47:37.311179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:37.311125 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:37.521548 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:37.521454 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:37.728082 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:37.728039 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:38.258570 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:38.258527 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 09:47:38.258745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:38.258601 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 09:47:38.548923 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:38.548806 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:38.743363 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:38.743325 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:39.401381 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:39.401344 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:39.402738 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:39.402716 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 09:47:39.859788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:39.859754 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:40.926521 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:40.926489 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:41.947055 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:41.947021 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:43.050627 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:43.050595 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:44.295011 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:44.294960 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:45.540326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:45.540275 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:45.873963 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:45.864592 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8"] Nov 02 09:47:46.045518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.007014 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:46.175735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.175646 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8"] Nov 02 09:47:46.176107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.176064 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.195494 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.195456 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-204fba-serving-cert\"" Nov 02 09:47:46.281636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.281595 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.282014 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.281987 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-metrics-certs\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.282197 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.282178 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.282397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.282375 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9d5\" (UniqueName: \"kubernetes.io/projected/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-kube-api-access-xb9d5\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.297217 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.297177 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:46.387795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.387750 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-metrics-certs\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.397916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.397863 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.408319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.408277 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9d5\" (UniqueName: \"kubernetes.io/projected/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-kube-api-access-xb9d5\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.410596 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.410568 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.412879 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.412853 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.533921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.533819 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.559638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.559597 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:46.569913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.566798 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-metrics-certs\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.569913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.567389 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9d5\" (UniqueName: \"kubernetes.io/projected/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-kube-api-access-xb9d5\") pod \"maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:46.762926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:46.761187 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:47.208265 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:47.208163 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:47:47.208541 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:47.208497 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:47:47.761969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:47.739816 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:47:47.761969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:47.739883 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:47:48.011691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:48.011643 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:48.020366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:48.018308 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8"] Nov 02 09:47:49.009250 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:49.009213 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" event={"ID":"75a15cbe-3d8e-4f96-abbc-e9f91a37596c","Type":"ContainerStarted","Data":"71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd"} Nov 02 09:47:49.009692 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:49.009277 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" event={"ID":"75a15cbe-3d8e-4f96-abbc-e9f91a37596c","Type":"ContainerStarted","Data":"352cd49035754a6f162624f2feea159cd6110167d39e6a93c5d8ce9ed7884822"} Nov 02 09:47:49.009692 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:49.009332 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:49.034129 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:49.034065 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" podStartSLOduration=3.518485454 podStartE2EDuration="4.034050546s" totalImagesPullingTime="515.565092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:49.033129102 +0000 UTC m=+747.873071096" watchObservedRunningTime="2026-11-02 09:47:49.034050546 +0000 UTC m=+747.873992541" Nov 02 09:47:49.519212 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:49.519172 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:50.389837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:50.389802 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:51.124175 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:51.124125 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:52.001076 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.001034 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:52.664333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.664293 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf"] Nov 02 09:47:52.668669 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.668640 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.686309 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.686277 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf"] Nov 02 09:47:52.773806 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.773763 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.774036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.773816 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.774036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.773930 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.774036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.773968 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.774036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.774024 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvsn\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-kube-api-access-zpvsn\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.774256 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.774117 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.774256 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.774172 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.774256 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.774205 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.774367 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.774331 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875372 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875428 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875468 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875495 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875519 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvsn\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-kube-api-access-zpvsn\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875549 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875601 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875634 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.875750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.875702 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.876749 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.876716 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.877363 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.877288 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.877488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.877353 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.877560 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.877528 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.877621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.877590 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.881449 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.881419 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.890519 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.890472 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvsn\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-kube-api-access-zpvsn\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.892154 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.892123 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.899326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.899290 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:52.904303 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:52.904271 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:53.012974 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:53.012848 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:53.303417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:53.303249 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf"] Nov 02 09:47:53.303567 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:47:53.303491 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7de95e_9a9d_4f12_b7a2_b0f84902240b.slice/crio-b139ab62575b2f86d2b5561f6786b09898f0c8d24faa94b30757891064cd1e50 WatchSource:0}: Error finding container b139ab62575b2f86d2b5561f6786b09898f0c8d24faa94b30757891064cd1e50: Status 404 returned error can't find the container with id b139ab62575b2f86d2b5561f6786b09898f0c8d24faa94b30757891064cd1e50 Nov 02 09:47:53.315966 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:53.307618 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:53.315966 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:53.307707 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:53.315966 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:53.307750 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:47:53.836908 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:53.836864 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:53.912910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:53.894206 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:47:54.049953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:54.049908 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" event={"ID":"4e7de95e-9a9d-4f12-b7a2-b0f84902240b","Type":"ContainerStarted","Data":"21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877"} Nov 02 09:47:54.050439 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:54.049972 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" event={"ID":"4e7de95e-9a9d-4f12-b7a2-b0f84902240b","Type":"ContainerStarted","Data":"b139ab62575b2f86d2b5561f6786b09898f0c8d24faa94b30757891064cd1e50"} Nov 02 09:47:55.020916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:55.014241 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:47:55.046924 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:55.036209 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 09:47:55.046924 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:55.036278 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 09:47:55.062920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:55.054058 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 09:47:55.089912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:55.084938 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podStartSLOduration=3.084917363 podStartE2EDuration="3.084917363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:54.106262395 +0000 UTC m=+752.946204391" watchObservedRunningTime="2026-11-02 09:47:55.084917363 +0000 UTC m=+753.924859361" Nov 02 09:47:55.924916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:55.924793 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:56.015336 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:56.015295 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 09:47:56.015648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:56.015606 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 09:47:57.014917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:57.014357 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 09:47:57.014917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:57.014449 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 09:47:57.754922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:57.752644 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:57.754922 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:47:57.752801 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:47:58.018981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:58.018218 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 09:47:58.018981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:58.018296 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 09:47:58.942316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:58.942279 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:47:59.015331 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:59.015294 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 09:47:59.015602 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:47:59.015575 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 09:48:00.013500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:00.013462 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 09:48:00.013945 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:00.013532 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 09:48:00.732647 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:00.732607 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:02.015182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:02.015138 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:02.055081 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:02.055043 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:48:02.055272 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:02.055133 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:48:02.241721 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:02.241674 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:48:02.242173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:02.242148 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:48:02.243300 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:02.243281 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:48:02.443745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:02.443707 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:03.118755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:03.118671 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:03.593082 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:03.593048 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:03.776373 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:03.776342 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:03.959535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:03.959444 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:04.243435 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:04.243335 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:04.431512 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:04.431470 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:05.137678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:05.137636 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:05.386178 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:05.386143 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:05.859120 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:05.859088 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:06.083616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:06.083579 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:06.932914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:06.930545 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:08.238019 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:08.237983 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:08.948494 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:08.948448 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:09.723514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:09.723477 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:10.451163 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:10.451123 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:10.769282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:10.769201 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:11.351648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:11.351611 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:12.085713 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:12.085679 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:12.985216 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:12.985181 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:13.851813 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:13.851770 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:14.743959 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:14.743922 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:14.996582 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:14.996488 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:48:15.724716 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:15.724680 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:17.510918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:17.508409 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:19.005402 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:19.005351 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:19.943213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:19.943153 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:20.756961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:20.756921 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:21.529992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:21.529948 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:22.255326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:22.255290 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:23.519607 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:23.519568 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:24.637876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:24.636651 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:25.463584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:25.463547 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:26.353498 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.353460 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs"] Nov 02 09:48:26.359574 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.359544 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.371484 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.371093 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9e7156-openshift-default-dockercfg-nlp5g\"" Nov 02 09:48:26.373547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.373516 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs"] Nov 02 09:48:26.390931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.390872 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:26.394229 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394196 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.394402 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394240 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.394473 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394401 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5x4\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-kube-api-access-rf5x4\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.394473 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394450 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.394584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394520 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.394584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394552 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.394703 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394668 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.394703 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394693 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.394872 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.394851 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496376 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496332 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496381 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496444 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5x4\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-kube-api-access-rf5x4\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496477 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496505 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496534 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496642 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496667 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.496823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496742 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.497045 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496807 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.497045 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.496820 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.497150 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.497066 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.497270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.497240 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.497455 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.497432 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.499316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.499297 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.499427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.499404 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.508914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.508867 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.509182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.509158 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5x4\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-kube-api-access-rf5x4\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.679664 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.679574 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:26.855480 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:48:26.855447 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:48:26.968639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.968606 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs"] Nov 02 09:48:26.970232 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:48:26.970204 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2107c6_8058_45a0_b71d_ebc0c908b9f2.slice/crio-0372e5dc6e7843f5cb37366f3e66122dc9525b3674a6155ce1292ce13f32bedc WatchSource:0}: Error finding container 0372e5dc6e7843f5cb37366f3e66122dc9525b3674a6155ce1292ce13f32bedc: Status 404 returned error can't find the container with id 0372e5dc6e7843f5cb37366f3e66122dc9525b3674a6155ce1292ce13f32bedc Nov 02 09:48:26.972639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.972605 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:48:26.972752 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.972681 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:48:26.972752 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:26.972731 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:48:27.159746 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:27.159702 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:27.765658 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:27.765623 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" event={"ID":"7f2107c6-8058-45a0-b71d-ebc0c908b9f2","Type":"ContainerStarted","Data":"869a83b65740ed8d8d23db3f6fa19ce3a7f8320796661fd77f4bcd1ec013f5df"} Nov 02 09:48:27.766203 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:27.765678 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" event={"ID":"7f2107c6-8058-45a0-b71d-ebc0c908b9f2","Type":"ContainerStarted","Data":"0372e5dc6e7843f5cb37366f3e66122dc9525b3674a6155ce1292ce13f32bedc"} Nov 02 09:48:27.811913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:27.810954 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" podStartSLOduration=1.810929862 podStartE2EDuration="1.810929862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:27.788636194 +0000 UTC m=+786.628578189" watchObservedRunningTime="2026-11-02 09:48:27.810929862 +0000 UTC m=+786.650871861" Nov 02 09:48:27.851232 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:27.851195 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:28.660652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:28.660622 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:28.680138 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:28.680108 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:28.681561 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:28.681535 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 09:48:28.681692 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:28.681595 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 09:48:29.445187 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:29.445111 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:29.680829 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:29.680789 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 09:48:29.681027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:29.680857 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 09:48:30.163628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:30.163597 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:30.680816 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:30.680772 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 09:48:30.681302 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:30.680835 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 09:48:30.853921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:30.853168 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:31.556422 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:31.556381 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:31.680472 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:31.680439 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 09:48:31.680647 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:31.680500 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 09:48:32.262772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:32.262733 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:32.680112 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:32.680080 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 09:48:32.680263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:32.680133 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 09:48:33.097319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:33.097276 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:33.518925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:33.518800 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:48:33.685620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:33.685589 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:33.685915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:33.685869 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:33.686666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:33.686649 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:48:33.794473 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:33.794383 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:35.068934 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:35.064849 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:36.859601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:36.859566 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:37.969343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:37.969305 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:38.881669 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:38.881636 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:39.731031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:39.730987 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:40.463802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:40.463770 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:41.158402 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:41.158300 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:41.859965 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:41.859927 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:42.568605 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:42.568567 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:43.613042 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:43.613009 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:43.953027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:43.952915 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s"] Nov 02 09:48:43.957809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:43.957775 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:43.965941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:43.965909 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-301be8-openshift-default-dockercfg-fxzjg\"" Nov 02 09:48:43.978349 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:43.973715 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s"] Nov 02 09:48:44.048818 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.048774 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5108180-72b0-4162-b798-e35bdd62825e-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.049063 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.048877 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.049063 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.048965 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.049227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.049110 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.049227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.049174 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.049361 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.049286 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.049427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.049384 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.049486 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.049422 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5108180-72b0-4162-b798-e35bdd62825e-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.049486 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.049447 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq96t\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-kube-api-access-dq96t\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.151481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.151418 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.151692 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.151604 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.151692 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.151662 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5108180-72b0-4162-b798-e35bdd62825e-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.151692 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.151687 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq96t\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-kube-api-access-dq96t\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.151882 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.151842 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5108180-72b0-4162-b798-e35bdd62825e-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.151882 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.151868 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.152025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.151929 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.152025 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.151971 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.152143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.152045 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.152143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.152077 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.152303 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.152280 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.152474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.152449 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.152566 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.152543 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.153481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.153458 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5108180-72b0-4162-b798-e35bdd62825e-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.155116 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.155097 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.155606 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.155580 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5108180-72b0-4162-b798-e35bdd62825e-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.168868 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.168832 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq96t\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-kube-api-access-dq96t\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.170972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.170943 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.306251 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.306132 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:44.570921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:44.568353 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:45.103100 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:45.102956 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s"] Nov 02 09:48:45.160349 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:45.148031 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:48:45.160349 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:45.148123 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:48:45.160349 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:45.148170 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:48:45.564522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:45.564481 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:46.091340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:46.091292 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" event={"ID":"c5108180-72b0-4162-b798-e35bdd62825e","Type":"ContainerStarted","Data":"228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e"} Nov 02 09:48:46.091542 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:46.091370 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" event={"ID":"c5108180-72b0-4162-b798-e35bdd62825e","Type":"ContainerStarted","Data":"ddb30b1623f7df5a1d61a8a429a5c2f2583a0c5ebfd3fba05787dd26ba83db81"} Nov 02 09:48:46.119388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:46.119322 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podStartSLOduration=3.119302148 podStartE2EDuration="3.119302148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:46.11778203 +0000 UTC m=+804.957724025" watchObservedRunningTime="2026-11-02 09:48:46.119302148 +0000 UTC m=+804.959244143" Nov 02 09:48:46.312911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:46.311247 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:46.331914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:46.327074 4467 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 09:48:46.331914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:46.327139 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 09:48:46.343016 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:46.335089 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:47.212925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:47.212407 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:47.308916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:47.308325 4467 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 09:48:47.308916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:47.308390 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 09:48:47.435037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:47.435001 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:47.643911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:47.643473 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:48.308985 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:48.308312 4467 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 09:48:48.308985 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:48.308377 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 09:48:48.481261 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:48.481227 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:49.263735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:49.263705 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:49.307282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:49.307238 4467 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 09:48:49.307491 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:49.307296 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 09:48:49.957664 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:49.957629 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:50.313916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:50.310350 4467 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 09:48:50.313916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:50.310424 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 09:48:50.787874 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:50.787773 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:51.307791 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:51.307332 4467 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 09:48:51.307791 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:51.307399 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 09:48:51.570628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:51.570534 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:52.362969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:52.362933 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:52.534881 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:52.534843 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:52.535768 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:52.535743 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:52.536693 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:52.536663 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:48:53.265987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:53.265953 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:54.115418 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:54.115367 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:54.872507 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:54.872460 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:55.591368 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:55.591334 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:56.087332 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:56.087300 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:48:56.297182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:56.297138 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:56.859183 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:48:56.859139 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:48:57.781264 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:57.776159 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:58.971547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:58.971496 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:48:59.905842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:48:59.905807 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:00.888282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:00.888244 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:01.763936 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:01.763884 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:02.599145 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:02.599077 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:03.485299 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:03.485255 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:04.187409 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:04.187364 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:04.962546 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:04.962501 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:05.676451 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:05.676415 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:06.391211 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:06.391173 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:07.908918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:07.902725 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:08.399031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.398962 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8"] Nov 02 09:49:08.461917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.450313 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.537919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.499062 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8743a5-openshift-default-dockercfg-6pdx6\"" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.568683 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.568741 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.568778 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafc909a-d460-45bf-b1b4-fc0748fd273f-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.568815 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.568862 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2s27\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-kube-api-access-z2s27\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.569027 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.569066 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.569098 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.569333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.569130 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.652002 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.646955 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8"] Nov 02 09:49:08.870922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.861656 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.915092 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.915365 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafc909a-d460-45bf-b1b4-fc0748fd273f-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.915421 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.915490 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2s27\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-kube-api-access-z2s27\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.915879 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.915959 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.915999 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.916039 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.914970 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.919543 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.920612 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafc909a-d460-45bf-b1b4-fc0748fd273f-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:08.951998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.948220 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:09.007917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.988427 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:09.007917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:08.989065 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:09.049969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:09.015558 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:09.111544 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:09.105578 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:09.116079 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:09.112861 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2s27\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-kube-api-access-z2s27\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:09.310928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:09.310160 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:09.382941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:09.376305 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:10.543980 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:10.540290 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:10.685740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:10.685700 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8"] Nov 02 09:49:10.798006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:10.782509 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:10.798006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:10.789334 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:49:10.798006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:10.789528 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:49:10.798006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:10.789644 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:49:10.970220 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:10.970170 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:11.430847 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:11.429600 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" event={"ID":"fafc909a-d460-45bf-b1b4-fc0748fd273f","Type":"ContainerStarted","Data":"23a170f5434c36a9683b40d5f5a417ff2ffb01a804943c7ea9dbd16f0e3d5e62"} Nov 02 09:49:11.430847 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:11.429683 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" event={"ID":"fafc909a-d460-45bf-b1b4-fc0748fd273f","Type":"ContainerStarted","Data":"127424fd83ebf313836742a9c3f2d900182be28d6b17a69a20bd523fb9169ae8"} Nov 02 09:49:11.458932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:11.457859 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" podStartSLOduration=3.457846661 podStartE2EDuration="3.457846661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:11.456978666 +0000 UTC m=+830.296920661" watchObservedRunningTime="2026-11-02 09:49:11.457846661 +0000 UTC m=+830.297788735" Nov 02 09:49:11.729959 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:11.729589 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:12.311283 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:12.311245 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:12.312754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:12.312720 4467 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 09:49:12.312907 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:12.312774 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 09:49:12.655322 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:12.655218 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:12.963677 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:12.963509 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:13.316400 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:13.316368 4467 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 09:49:13.316673 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:13.316648 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 09:49:13.942174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:13.942136 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:14.314085 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:14.313335 4467 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 09:49:14.314085 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:14.313402 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 09:49:14.817972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:14.816811 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:15.311545 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:15.311511 4467 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 09:49:15.311750 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:15.311575 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 09:49:15.669034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:15.668924 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:16.316549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:16.316030 4467 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 09:49:16.316549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:16.316100 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 09:49:16.351134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:16.351094 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:17.101917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:17.096910 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:17.311926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:17.311873 4467 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 09:49:17.312124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:17.311948 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 09:49:17.836339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:17.836307 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:18.245346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:18.245257 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:18.365596 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:18.365570 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:18.366404 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:18.366352 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:18.369022 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:18.368591 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:49:18.885641 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:18.885611 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:19.571457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:19.571420 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:49:19.671471 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:19.671436 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:20.582920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:20.582377 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:20.845725 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:20.845632 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:22.750180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:22.750137 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:23.851270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:23.851227 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:24.671045 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:24.671004 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:25.635981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:25.635946 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:26.456502 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:26.456457 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:26.855317 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:49:26.855278 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:49:27.345490 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:27.345451 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:28.064661 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:28.064619 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:28.748805 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:28.748763 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:29.453319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:29.453289 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:30.155694 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:30.155656 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:30.960229 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:30.960193 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:31.664489 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:31.664407 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:32.916405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:32.916369 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:34.785920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:34.765560 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:36.149159 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:36.149120 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:36.985766 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:36.985675 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:37.853952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:37.853882 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:38.658794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:38.658749 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:39.471634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:39.471604 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:39.668586 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:39.668543 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:40.295786 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:40.295751 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:40.483219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:40.483145 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:41.295727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:41.295681 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:42.089528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:42.089464 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:43.030297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:43.030266 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:43.875306 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:43.875269 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:45.576567 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:45.576532 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:49:45.774727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:45.774691 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:47.265424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:47.265386 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:47.872098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:47.872061 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:49:47.872456 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:47.872423 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:49:48.292438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:48.292038 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:49:48.292438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:48.292105 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:49:49.059476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:49.059407 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:49.632535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:49.632489 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:51.018236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:51.018181 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:51.223158 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:51.223125 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:51.990152 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:51.990106 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:52.242410 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:52.242327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:53.310675 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:53.310643 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:53.533006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:53.532963 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:53.751575 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:53.751469 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:54.557270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:54.557232 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:55.352166 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:55.352132 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:55.586600 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:55.586559 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:55.768002 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:55.767864 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:56.264546 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:56.264505 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:56.856471 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:49:56.856431 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:49:56.967950 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:56.967918 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:57.680490 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:57.680453 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:58.315182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:58.315142 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:58.501764 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:58.501720 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:59.103350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:59.103293 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:49:59.594640 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:49:59.594608 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:00.151282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:00.151247 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:00.899159 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:00.899127 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:01.243162 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:01.243072 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:01.461365 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:01.461327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:02.454926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:02.454708 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:02.708623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:02.708530 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:03.286789 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:03.286750 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:03.480389 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:03.480354 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:03.715366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:03.715326 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:04.043973 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:04.043863 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:04.360883 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:04.360843 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:04.570858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:04.570823 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:05.060605 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:05.060571 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:05.293177 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:05.293129 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:06.317186 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:06.317156 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:06.895521 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:06.895482 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:07.092228 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:07.092192 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:07.360638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:07.360605 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:08.083188 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:08.083141 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:08.440549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:08.440460 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:08.853801 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:08.853760 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:09.552797 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:09.552758 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:10.349616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:10.349583 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:10.881731 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:10.881695 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:11.069827 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:11.069795 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:11.568668 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:11.568629 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps"] Nov 02 09:50:11.569007 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:11.568953 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" podUID="2f3ad92b-69c8-4e04-9203-a89302aa146e" containerName="maas-api" containerID="cri-o://0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d" gracePeriod=30 Nov 02 09:50:11.711478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:11.711447 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:11.995733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:11.995708 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:50:12.061462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.060446 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3ad92b-69c8-4e04-9203-a89302aa146e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ad92b-69c8-4e04-9203-a89302aa146e-openshift-service-ca\") pod \"2f3ad92b-69c8-4e04-9203-a89302aa146e\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " Nov 02 09:50:12.061879 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.061406 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3ad92b-69c8-4e04-9203-a89302aa146e-openshift-service-ca" pod "2f3ad92b-69c8-4e04-9203-a89302aa146e" (UID: "2f3ad92b-69c8-4e04-9203-a89302aa146e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:50:12.062154 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.062135 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-metrics-certs\") pod \"2f3ad92b-69c8-4e04-9203-a89302aa146e\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " Nov 02 09:50:12.062378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.062360 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-maas-api-tls\") pod \"2f3ad92b-69c8-4e04-9203-a89302aa146e\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " Nov 02 09:50:12.062544 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.062531 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3ad92b-69c8-4e04-9203-a89302aa146e-kube-api-access-99rpg\" (UniqueName: \"kubernetes.io/projected/2f3ad92b-69c8-4e04-9203-a89302aa146e-kube-api-access-99rpg\") pod \"2f3ad92b-69c8-4e04-9203-a89302aa146e\" (UID: \"2f3ad92b-69c8-4e04-9203-a89302aa146e\") " Nov 02 09:50:12.063273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.063231 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ad92b-69c8-4e04-9203-a89302aa146e-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:12.064748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.064704 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:12.066937 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.066236 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3ad92b-69c8-4e04-9203-a89302aa146e-kube-api-access-99rpg" pod "2f3ad92b-69c8-4e04-9203-a89302aa146e" (UID: "2f3ad92b-69c8-4e04-9203-a89302aa146e"). InnerVolumeSpecName "kube-api-access-99rpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:50:12.066937 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.066439 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-metrics-certs" pod "2f3ad92b-69c8-4e04-9203-a89302aa146e" (UID: "2f3ad92b-69c8-4e04-9203-a89302aa146e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:50:12.068294 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.068270 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-maas-api-tls" pod "2f3ad92b-69c8-4e04-9203-a89302aa146e" (UID: "2f3ad92b-69c8-4e04-9203-a89302aa146e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:50:12.164426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.164375 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:12.164426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.164420 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f3ad92b-69c8-4e04-9203-a89302aa146e-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:12.164426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.164431 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99rpg\" (UniqueName: \"kubernetes.io/projected/2f3ad92b-69c8-4e04-9203-a89302aa146e-kube-api-access-99rpg\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:12.544307 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.544268 4467 generic.go:356] "Generic (PLEG): container finished" podID="2f3ad92b-69c8-4e04-9203-a89302aa146e" containerID="0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d" exitCode=0 Nov 02 09:50:12.544528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.544309 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" event={"ID":"2f3ad92b-69c8-4e04-9203-a89302aa146e","Type":"ContainerDied","Data":"0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d"} Nov 02 09:50:12.544528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.544330 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" Nov 02 09:50:12.544528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.544379 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps" event={"ID":"2f3ad92b-69c8-4e04-9203-a89302aa146e","Type":"ContainerDied","Data":"858a858225bd50eff46d30e80da9738ae4ba9534ea0f36bcd948a1d5d1bcf148"} Nov 02 09:50:12.544528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.544408 4467 scope.go:122] "RemoveContainer" containerID="0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d" Nov 02 09:50:12.555258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.555244 4467 scope.go:122] "RemoveContainer" containerID="0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d" Nov 02 09:50:12.555523 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:50:12.555507 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d\": container with ID starting with 0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d not found: ID does not exist" containerID="0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d" Nov 02 09:50:12.555591 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.555530 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d"} err="failed to get container status \"0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d\": rpc error: code = NotFound desc = could not find container \"0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d\": container with ID starting with 0e1e478c591b0ddb5c459632dc8cbea307ed54a85fac5af81b8f10413448c16d not found: ID does not exist" Nov 02 09:50:12.570784 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.570759 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps"] Nov 02 09:50:12.577009 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.576976 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-301be8-9bdc9f9cf-xv4ps"] Nov 02 09:50:12.942246 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:12.942213 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:13.333067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:13.333026 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3ad92b-69c8-4e04-9203-a89302aa146e" path="/var/lib/kubelet/pods/2f3ad92b-69c8-4e04-9203-a89302aa146e/volumes" Nov 02 09:50:13.657741 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:13.657652 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:13.830188 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:13.830152 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:50:14.665930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:14.665878 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:15.640497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:15.640461 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:15.854035 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:15.853990 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:17.009914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.009297 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:50:17.009914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.009407 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:50:17.214926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.214042 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:50:17.214926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.214127 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:50:17.298976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.294427 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:17.731530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.731009 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:50:17.731530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.731101 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:50:17.987329 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.987231 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s"] Nov 02 09:50:17.987864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:17.987829 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" containerID="cri-o://228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e" gracePeriod=30 Nov 02 09:50:18.026133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:18.026096 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:50:18.036196 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:18.036148 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:50:18.037128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:18.037090 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:18.320050 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:18.320004 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:50:18.320278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:18.320081 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:50:18.819418 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:18.819364 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:19.829885 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:19.829809 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:20.655480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:20.655443 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:21.408506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:21.408389 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:22.152564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:22.152530 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:22.538131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:22.538096 4467 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 09:50:22.538514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:22.538162 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 09:50:22.970944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:22.970879 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:23.374596 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.374568 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:50:23.476737 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.476696 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-credential-socket\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.476948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.476745 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-socket\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.476948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.476845 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-data\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.476948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.476917 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5108180-72b0-4162-b798-e35bdd62825e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5108180-72b0-4162-b798-e35bdd62825e-istiod-ca-cert\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.477140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.476992 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-envoy\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.477140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477009 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-credential-socket" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:23.477140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477023 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-certs\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.477140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477060 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5108180-72b0-4162-b798-e35bdd62825e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5108180-72b0-4162-b798-e35bdd62825e-istio-podinfo\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.477140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477090 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-socket" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:23.477462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477156 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-istio-token\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-istio-token\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.477462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477217 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-data" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:23.477462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477234 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-kube-api-access-dq96t\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-kube-api-access-dq96t\") pod \"c5108180-72b0-4162-b798-e35bdd62825e\" (UID: \"c5108180-72b0-4162-b798-e35bdd62825e\") " Nov 02 09:50:23.477462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477306 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-certs" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:23.477462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477316 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5108180-72b0-4162-b798-e35bdd62825e-istiod-ca-cert" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:50:23.477741 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477717 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.477873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477744 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.477873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477760 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.477873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477772 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5108180-72b0-4162-b798-e35bdd62825e-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.477873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.477781 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.479455 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.479422 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-kube-api-access-dq96t" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "kube-api-access-dq96t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:50:23.479455 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.479432 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5108180-72b0-4162-b798-e35bdd62825e-istio-podinfo" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:50:23.479625 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.479605 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-istio-token" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:50:23.479683 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.479665 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-envoy" pod "c5108180-72b0-4162-b798-e35bdd62825e" (UID: "c5108180-72b0-4162-b798-e35bdd62825e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:23.578804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.578760 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5108180-72b0-4162-b798-e35bdd62825e-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.578804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.578796 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5108180-72b0-4162-b798-e35bdd62825e-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.578804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.578810 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.579319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.578822 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq96t\" (UniqueName: \"kubernetes.io/projected/c5108180-72b0-4162-b798-e35bdd62825e-kube-api-access-dq96t\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:23.732690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.732657 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:23.765350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.765303 4467 generic.go:356] "Generic (PLEG): container finished" podID="c5108180-72b0-4162-b798-e35bdd62825e" containerID="228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e" exitCode=0 Nov 02 09:50:23.765350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.765349 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" Nov 02 09:50:23.765557 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.765349 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" event={"ID":"c5108180-72b0-4162-b798-e35bdd62825e","Type":"ContainerDied","Data":"228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e"} Nov 02 09:50:23.765557 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.765480 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s" event={"ID":"c5108180-72b0-4162-b798-e35bdd62825e","Type":"ContainerDied","Data":"ddb30b1623f7df5a1d61a8a429a5c2f2583a0c5ebfd3fba05787dd26ba83db81"} Nov 02 09:50:23.765557 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.765507 4467 scope.go:122] "RemoveContainer" containerID="228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e" Nov 02 09:50:23.775732 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.775714 4467 scope.go:122] "RemoveContainer" containerID="228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e" Nov 02 09:50:23.776085 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:50:23.776066 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e\": container with ID starting with 228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e not found: ID does not exist" containerID="228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e" Nov 02 09:50:23.776146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.776095 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e"} err="failed to get container status \"228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e\": rpc error: code = NotFound desc = could not find container \"228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e\": container with ID starting with 228352933c77072253e60e6fa9780e86b0eead4da19d54f5669b190cc412ee1e not found: ID does not exist" Nov 02 09:50:23.792416 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.792385 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s"] Nov 02 09:50:23.796740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:23.796713 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-cxz8s"] Nov 02 09:50:24.450908 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:24.450846 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:25.026492 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:25.026455 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:25.207483 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:25.207441 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:25.341636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:25.341597 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5108180-72b0-4162-b798-e35bdd62825e" path="/var/lib/kubelet/pods/c5108180-72b0-4162-b798-e35bdd62825e/volumes" Nov 02 09:50:25.403542 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:25.403509 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:25.933799 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:25.933766 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:26.116712 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:26.116651 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:27.229918 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:50:27.226109 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:50:28.624918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:28.622125 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:30.138930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:30.124540 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:31.216596 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:31.216556 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:31.951085 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:31.951051 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:32.173282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.173236 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:32.399488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.399455 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58"] Nov 02 09:50:32.400832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.400796 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" podUID="6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" containerName="maas-api" containerID="cri-o://c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf" gracePeriod=30 Nov 02 09:50:32.540660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.540629 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:32.782926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.782863 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:50:32.812840 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.812807 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls\") pod \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " Nov 02 09:50:32.813089 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.812964 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-openshift-service-ca\") pod \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " Nov 02 09:50:32.813089 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.813011 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-kube-api-access-z5pk7\" (UniqueName: \"kubernetes.io/projected/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-kube-api-access-z5pk7\") pod \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " Nov 02 09:50:32.813214 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.813112 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-metrics-certs\") pod \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\" (UID: \"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9\") " Nov 02 09:50:32.814259 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.813973 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-openshift-service-ca" pod "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" (UID: "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:50:32.819942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.816845 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:32.823457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.823429 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-kube-api-access-z5pk7" pod "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" (UID: "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9"). InnerVolumeSpecName "kube-api-access-z5pk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:50:32.823588 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.823561 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls" pod "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" (UID: "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:50:32.826076 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.826048 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-metrics-certs" pod "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" (UID: "6903cdc1-3ed6-4ff1-9225-096d1d7e65a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:50:32.886077 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.886032 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:32.919849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.919787 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5pk7\" (UniqueName: \"kubernetes.io/projected/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-kube-api-access-z5pk7\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:32.920223 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.920205 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:32.920327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:32.920308 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:33.219074 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219031 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq"] Nov 02 09:50:33.219597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219585 4467 cpu_manager.go:414] "containerMap: removing container" podUID="6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" containerName="maas-api" Nov 02 09:50:33.219642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219601 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" containerName="maas-api" Nov 02 09:50:33.219642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219620 4467 cpu_manager.go:414] "containerMap: removing container" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" Nov 02 09:50:33.219642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219626 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" Nov 02 09:50:33.219642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219636 4467 cpu_manager.go:414] "containerMap: removing container" podUID="2f3ad92b-69c8-4e04-9203-a89302aa146e" containerName="maas-api" Nov 02 09:50:33.219642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219641 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3ad92b-69c8-4e04-9203-a89302aa146e" containerName="maas-api" Nov 02 09:50:33.219792 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219723 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5108180-72b0-4162-b798-e35bdd62825e" containerName="istio-proxy" Nov 02 09:50:33.219792 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219732 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3ad92b-69c8-4e04-9203-a89302aa146e" containerName="maas-api" Nov 02 09:50:33.219792 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.219743 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" containerName="maas-api" Nov 02 09:50:33.223263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.223240 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.227660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.227638 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-204fba-openshift-default-dockercfg-xvjxq\"" Nov 02 09:50:33.246549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.245337 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq"] Nov 02 09:50:33.316970 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.316931 4467 generic.go:356] "Generic (PLEG): container finished" podID="6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" containerID="c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf" exitCode=0 Nov 02 09:50:33.317142 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.317002 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" Nov 02 09:50:33.317142 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.317016 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" event={"ID":"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9","Type":"ContainerDied","Data":"c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf"} Nov 02 09:50:33.317142 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.317083 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58" event={"ID":"6903cdc1-3ed6-4ff1-9225-096d1d7e65a9","Type":"ContainerDied","Data":"8c364412ef678d40397cf6927f82a19fb0ae93338fc5c6fd26bf9bfe727f2f08"} Nov 02 09:50:33.317142 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.317107 4467 scope.go:122] "RemoveContainer" containerID="c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf" Nov 02 09:50:33.326069 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326038 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326096 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326272 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326208 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326272 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326252 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr252\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-kube-api-access-sr252\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326281 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3264c12-364d-44ab-a875-d19ae898b5ab-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326338 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326445 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326563 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326547 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326618 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326577 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.326761 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.326745 4467 scope.go:122] "RemoveContainer" containerID="c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf" Nov 02 09:50:33.327099 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:50:33.327081 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf\": container with ID starting with c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf not found: ID does not exist" containerID="c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf" Nov 02 09:50:33.327184 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.327106 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf"} err="failed to get container status \"c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf\": rpc error: code = NotFound desc = could not find container \"c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf\": container with ID starting with c3907e286f69dfde1f129a839e743e494f81be5178611a8c64593503211c92cf not found: ID does not exist" Nov 02 09:50:33.343524 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.343490 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58"] Nov 02 09:50:33.348091 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.347651 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8743a5-5bc69d8fdc-qbr58"] Nov 02 09:50:33.427693 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.427644 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.427706 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.427728 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr252\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-kube-api-access-sr252\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.427746 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3264c12-364d-44ab-a875-d19ae898b5ab-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.427770 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.428203 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428484 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.428391 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428540 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.428527 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428593 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.428560 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428650 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.428588 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3264c12-364d-44ab-a875-d19ae898b5ab-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.428650 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428773 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.428746 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.428826 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.428807 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.429095 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.429055 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.430430 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.430409 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.430790 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.430770 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.440321 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.440285 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.440426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.440354 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr252\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-kube-api-access-sr252\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.550582 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.550498 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:33.552525 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.552495 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:33.834958 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.834909 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq"] Nov 02 09:50:33.837240 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:50:33.837213 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3264c12_364d_44ab_a875_d19ae898b5ab.slice/crio-bd6f9c81dbb0239e0cbf20a836c07fcc57e67e1ab1de0dbe60ad47000e808f0c WatchSource:0}: Error finding container bd6f9c81dbb0239e0cbf20a836c07fcc57e67e1ab1de0dbe60ad47000e808f0c: Status 404 returned error can't find the container with id bd6f9c81dbb0239e0cbf20a836c07fcc57e67e1ab1de0dbe60ad47000e808f0c Nov 02 09:50:33.840559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.840525 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:50:33.840671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.840598 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:50:33.840671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:33.840628 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:50:34.242215 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:34.242118 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:34.332773 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:34.332733 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" event={"ID":"c3264c12-364d-44ab-a875-d19ae898b5ab","Type":"ContainerStarted","Data":"cfbd4c7b2b382eee245272815afc9c3eb937cc28e3fcfe97f73b81be7e4186bd"} Nov 02 09:50:34.333081 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:34.333061 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" event={"ID":"c3264c12-364d-44ab-a875-d19ae898b5ab","Type":"ContainerStarted","Data":"bd6f9c81dbb0239e0cbf20a836c07fcc57e67e1ab1de0dbe60ad47000e808f0c"} Nov 02 09:50:34.368035 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:34.367979 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podStartSLOduration=1.367962943 podStartE2EDuration="1.367962943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:34.364857581 +0000 UTC m=+913.204799575" watchObservedRunningTime="2026-11-02 09:50:34.367962943 +0000 UTC m=+913.207904938" Nov 02 09:50:34.551346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:34.551255 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:34.552766 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:34.552735 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 09:50:34.552900 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:34.552785 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 09:50:34.911546 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:34.911384 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:35.332591 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:35.332561 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6903cdc1-3ed6-4ff1-9225-096d1d7e65a9" path="/var/lib/kubelet/pods/6903cdc1-3ed6-4ff1-9225-096d1d7e65a9/volumes" Nov 02 09:50:35.551987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:35.551952 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 09:50:35.552388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:35.552022 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 09:50:35.573495 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:35.573457 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:35.781474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:35.781388 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:35.981342 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:35.981311 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:36.552990 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:36.552923 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 09:50:36.553513 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:36.552987 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 09:50:36.576099 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:36.576062 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:37.516574 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:37.516533 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:37.552028 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:37.551995 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 09:50:37.552244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:37.552081 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 09:50:38.350260 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:38.350216 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:38.537035 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:38.536996 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:38.551209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:38.551174 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 09:50:38.551380 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:38.551236 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 09:50:38.817951 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:38.817906 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:50:38.954140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:38.954100 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:39.201622 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:39.201525 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8"] Nov 02 09:50:39.220313 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:39.219211 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" containerID="cri-o://23a170f5434c36a9683b40d5f5a417ff2ffb01a804943c7ea9dbd16f0e3d5e62" gracePeriod=30 Nov 02 09:50:39.318411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:39.318378 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:39.518974 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:39.518517 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:39.552438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:39.552403 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 09:50:39.552755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:39.552724 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 09:50:40.422925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:40.422032 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:40.551645 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:40.551608 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 09:50:40.552339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:40.552288 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 09:50:41.627632 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:41.606695 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:41.627632 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:41.608140 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:41.627632 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:41.614259 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 09:50:41.709733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:41.709605 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:42.075976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:42.075922 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:42.273843 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:42.273057 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:42.694933 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:42.691356 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:42.872338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:42.872309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:43.545478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:43.545448 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:44.355939 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.355882 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:44.495425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.490048 4467 generic.go:356] "Generic (PLEG): container finished" podID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerID="23a170f5434c36a9683b40d5f5a417ff2ffb01a804943c7ea9dbd16f0e3d5e62" exitCode=0 Nov 02 09:50:44.495425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.490128 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" event={"ID":"fafc909a-d460-45bf-b1b4-fc0748fd273f","Type":"ContainerDied","Data":"23a170f5434c36a9683b40d5f5a417ff2ffb01a804943c7ea9dbd16f0e3d5e62"} Nov 02 09:50:44.515362 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.515332 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:50:44.645641 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.645611 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-certs\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.645822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.645658 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-envoy\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.645822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.645687 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-kube-api-access-z2s27\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-kube-api-access-z2s27\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.645822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.645749 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafc909a-d460-45bf-b1b4-fc0748fd273f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafc909a-d460-45bf-b1b4-fc0748fd273f-istiod-ca-cert\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.645822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.645777 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-token\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-token\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.646060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.645863 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-podinfo\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.646060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.645941 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-data\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.646060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.645982 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-certs" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:44.646060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646039 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-socket\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.646266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646066 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-credential-socket\") pod \"fafc909a-d460-45bf-b1b4-fc0748fd273f\" (UID: \"fafc909a-d460-45bf-b1b4-fc0748fd273f\") " Nov 02 09:50:44.646266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646242 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-data" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:44.646266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646255 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-socket" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:44.646455 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646420 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafc909a-d460-45bf-b1b4-fc0748fd273f-istiod-ca-cert" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:50:44.646566 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646538 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-credential-socket" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:44.646746 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646719 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:44.646859 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646749 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:44.646859 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646764 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:44.646859 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646777 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:44.646859 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.646792 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fafc909a-d460-45bf-b1b4-fc0748fd273f-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:44.648124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.648102 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-envoy" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:50:44.648219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.648157 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-kube-api-access-z2s27" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "kube-api-access-z2s27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:50:44.648370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.648348 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-token" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:50:44.648666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.648630 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-podinfo" pod "fafc909a-d460-45bf-b1b4-fc0748fd273f" (UID: "fafc909a-d460-45bf-b1b4-fc0748fd273f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:50:44.747921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.747825 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:44.747921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.747867 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:44.747921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.747882 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2s27\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-kube-api-access-z2s27\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:44.747921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:44.747914 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fafc909a-d460-45bf-b1b4-fc0748fd273f-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:50:45.139235 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:45.138203 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:45.497815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:45.497720 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" event={"ID":"fafc909a-d460-45bf-b1b4-fc0748fd273f","Type":"ContainerDied","Data":"127424fd83ebf313836742a9c3f2d900182be28d6b17a69a20bd523fb9169ae8"} Nov 02 09:50:45.497815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:45.497757 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8" Nov 02 09:50:45.497815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:45.497801 4467 scope.go:122] "RemoveContainer" containerID="23a170f5434c36a9683b40d5f5a417ff2ffb01a804943c7ea9dbd16f0e3d5e62" Nov 02 09:50:45.518848 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:45.518822 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8"] Nov 02 09:50:45.523771 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:45.523744 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-mv6t8"] Nov 02 09:50:45.860278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:45.860246 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:46.567488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:46.567452 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:47.273358 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:47.273321 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:47.333050 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:47.333018 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" path="/var/lib/kubelet/pods/fafc909a-d460-45bf-b1b4-fc0748fd273f/volumes" Nov 02 09:50:47.960922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:47.960819 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:48.760323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:48.760269 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:49.010762 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:49.010678 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:49.287391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:49.287299 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:49.589884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:49.589848 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:49.816835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:49.816795 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:50.453836 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:50.453783 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:51.076220 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:51.076175 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:50:51.205630 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:51.205592 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:52.394920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:52.394709 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:52.607091 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:52.607055 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:53.770691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:53.770644 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:54.177350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:54.177311 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:54.380142 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:54.379766 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:54.896315 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:54.896266 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:55.823409 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:55.823370 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:56.563054 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:56.563025 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:56.855699 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:50:56.855610 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:50:57.250468 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:57.250386 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:58.032629 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:58.032590 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:58.227806 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:58.227770 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:58.917628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:58.917590 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:59.144806 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:59.144774 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:50:59.324377 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:50:59.324327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:00.150655 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:00.150604 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:01.270149 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:01.270110 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:02.281880 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:02.281787 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:02.944143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:02.944108 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:03.652378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:03.652345 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:03.943826 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:03.943744 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:04.121864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:04.121830 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:04.724613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:04.724577 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:05.552536 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:05.552497 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:06.079309 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:06.079271 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:06.443310 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:06.443215 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:07.160447 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:07.160409 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:08.023060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:08.023022 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:08.758214 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:08.758180 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:09.593504 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:09.593469 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:10.265480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:10.265428 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:10.935278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:10.935239 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:11.120851 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:11.120820 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:11.833467 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:11.833416 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:11.961877 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:11.961840 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:51:12.648926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:12.648878 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:12.987547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:12.987441 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:13.184629 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:13.184593 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:15.983917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:15.983783 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:17.088915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:17.069310 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:51:17.088915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:17.069391 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:51:17.212912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:17.212004 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:51:17.212912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:17.212090 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:51:17.932803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:17.932754 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:51:17.933120 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:17.933089 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:51:18.254724 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.254644 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:18.294609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.294567 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr"] Nov 02 09:51:18.295406 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.295378 4467 cpu_manager.go:414] "containerMap: removing container" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" Nov 02 09:51:18.295576 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.295556 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" Nov 02 09:51:18.295747 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.295733 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafc909a-d460-45bf-b1b4-fc0748fd273f" containerName="istio-proxy" Nov 02 09:51:18.299320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.299298 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.313832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.312944 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr"] Nov 02 09:51:18.462733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462386 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.462733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462440 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.462733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462499 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.462733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462537 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.462733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462560 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.462733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462614 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.462733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462674 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.462733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462713 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.463312 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.462795 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7q8\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-kube-api-access-bx7q8\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564052 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564018 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7q8\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-kube-api-access-bx7q8\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564074 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564107 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564166 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564203 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564227 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564277 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564336 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564382 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564761 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:18.564941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564870 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.564941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564882 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.565055 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.564944 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.565179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.565152 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.565511 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.565493 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.567423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.567399 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.567686 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.567667 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.576987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.576955 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7q8\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-kube-api-access-bx7q8\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.577107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.576963 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.617194 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.617166 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:18.745444 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.745397 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:18.976910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.959263 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr"] Nov 02 09:51:18.980882 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:51:18.980846 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2cfa19_5036_404b_8f71_df1bb7c37b45.slice/crio-033f4333e5330fc47aad1aa0b2cd0291583a0ef75c0fef98dd20f4847f0b6560 WatchSource:0}: Error finding container 033f4333e5330fc47aad1aa0b2cd0291583a0ef75c0fef98dd20f4847f0b6560: Status 404 returned error can't find the container with id 033f4333e5330fc47aad1aa0b2cd0291583a0ef75c0fef98dd20f4847f0b6560 Nov 02 09:51:18.984383 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.984340 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:51:18.984507 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.984435 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:51:18.984507 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:18.984474 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:51:19.194229 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:19.194193 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" event={"ID":"4d2cfa19-5036-404b-8f71-df1bb7c37b45","Type":"ContainerStarted","Data":"acbc2427dbf193a31f16d19a08bee59637c8a08f59a309e3766768df21ed53f7"} Nov 02 09:51:19.194391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:19.194259 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" event={"ID":"4d2cfa19-5036-404b-8f71-df1bb7c37b45","Type":"ContainerStarted","Data":"033f4333e5330fc47aad1aa0b2cd0291583a0ef75c0fef98dd20f4847f0b6560"} Nov 02 09:51:19.219500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:19.219407 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" podStartSLOduration=1.21939377 podStartE2EDuration="1.21939377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:19.21753954 +0000 UTC m=+958.057481536" watchObservedRunningTime="2026-11-02 09:51:19.21939377 +0000 UTC m=+958.059335764" Nov 02 09:51:19.618060 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:19.618026 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:19.901979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:19.901079 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:20.529839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:20.529801 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:20.620218 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:20.620181 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 09:51:20.620606 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:20.620237 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 09:51:21.158484 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:21.158444 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:21.617719 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:21.617691 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 09:51:21.617901 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:21.617768 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 09:51:21.971166 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:21.971129 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:22.618476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:22.618439 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 09:51:22.618691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:22.618496 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 09:51:22.771098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:22.771048 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:23.612580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:23.612543 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:23.618312 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:23.618284 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 09:51:23.618516 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:23.618491 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 09:51:24.492920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:24.486416 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:24.618236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:24.618200 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 09:51:24.618633 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:24.618257 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 09:51:25.367379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:25.367336 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:25.622162 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:25.622084 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:25.622634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:25.622609 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:25.623559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:25.623537 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 09:51:26.288052 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:26.288013 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:26.996930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:26.996325 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:27.046265 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:51:27.046222 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:51:27.201483 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:27.201445 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:27.476357 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:27.476319 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:28.453918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:28.398457 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:28.676951 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:28.649760 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:29.783675 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:29.783611 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:30.202923 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:30.200409 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:30.445920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:30.445872 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:32.019573 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:32.019028 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:51:32.019573 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:32.019101 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:51:33.040588 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:33.040544 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:33.605970 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:33.605920 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:34.024418 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:34.024307 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:34.213157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:34.213123 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:35.135688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:35.135651 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:36.146691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:36.146655 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:36.845391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:36.845350 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:37.314731 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:37.314696 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:37.505280 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:37.505240 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:37.693173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:37.693085 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:37.923522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:37.923484 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:38.141808 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:38.141768 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:38.354254 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:38.354223 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:38.547141 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:38.547063 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:38.732793 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:38.732751 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:39.039974 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:39.039937 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:39.228506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:39.228473 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:39.427574 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:39.427447 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:39.731872 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:39.731723 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:40.078071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:40.078040 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:51:40.563788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:40.563727 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:42.677209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:42.677171 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:42.857456 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:42.857417 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:43.308749 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:43.308715 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:43.499911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:43.492942 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:44.253492 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:44.253454 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:45.232724 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.232691 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:45.794870 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.794837 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd"] Nov 02 09:51:45.800805 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.800780 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:45.810922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.810849 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9cbd331c-serving-cert\"" Nov 02 09:51:45.823427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.822385 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd"] Nov 02 09:51:45.932925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.932871 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-maas-api-tls\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:45.933101 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.933000 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-metrics-certs\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:45.933101 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.933057 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-openshift-service-ca\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:45.933185 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.933159 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9xg\" (UniqueName: \"kubernetes.io/projected/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-kube-api-access-qx9xg\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:45.939293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:45.939258 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:46.034450 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.034418 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9xg\" (UniqueName: \"kubernetes.io/projected/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-kube-api-access-qx9xg\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.034821 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.034794 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-maas-api-tls\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.035027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.035010 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-metrics-certs\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.035170 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.035081 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-openshift-service-ca\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.035979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.035957 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-openshift-service-ca\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.038182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.038155 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-metrics-certs\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.051577 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.051510 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-maas-api-tls\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.063489 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.063425 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9xg\" (UniqueName: \"kubernetes.io/projected/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-kube-api-access-qx9xg\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.116009 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.115976 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:46.133107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.133082 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:46.452386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.452347 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd"] Nov 02 09:51:46.457691 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:51:46.457664 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb7095c_ce6c_4164_a4d5_d80b62ccc9de.slice/crio-2d439a57a4cf834c36341f45077ca5ef2c172901f34b91d0cae2173edb2b8e20 WatchSource:0}: Error finding container 2d439a57a4cf834c36341f45077ca5ef2c172901f34b91d0cae2173edb2b8e20: Status 404 returned error can't find the container with id 2d439a57a4cf834c36341f45077ca5ef2c172901f34b91d0cae2173edb2b8e20 Nov 02 09:51:46.459510 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.459486 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 09:51:46.848549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:46.848505 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:47.079628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:47.079593 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" event={"ID":"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de","Type":"ContainerStarted","Data":"b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440"} Nov 02 09:51:47.079827 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:47.079659 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:47.079827 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:47.079679 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" event={"ID":"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de","Type":"ContainerStarted","Data":"2d439a57a4cf834c36341f45077ca5ef2c172901f34b91d0cae2173edb2b8e20"} Nov 02 09:51:47.102301 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:47.102209 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" podStartSLOduration=1.597250032 podStartE2EDuration="2.102195576s" totalImagesPullingTime="504.945544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:47.099492709 +0000 UTC m=+985.939434698" watchObservedRunningTime="2026-11-02 09:51:47.102195576 +0000 UTC m=+985.942137570" Nov 02 09:51:47.308882 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:47.308745 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:47.828730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:47.828691 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:48.185872 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:48.185771 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:48.657215 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:48.657183 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:49.444923 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:49.443345 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:50.256422 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:50.253588 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:50.434095 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:50.434061 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:51.140681 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:51.140629 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:51.935028 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:51.934992 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:52.782919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:52.782174 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:53.097191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:53.097153 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:51:53.869491 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:53.869453 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:54.103740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:54.103695 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:56.108144 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:56.108101 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:56.855686 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:51:56.855649 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:51:57.055584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:57.055554 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:57.884541 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:57.884479 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:58.066391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:58.066359 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:58.762959 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:58.762921 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:51:59.460916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:51:59.460865 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:00.208209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:00.208170 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:00.421922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:00.416033 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:00.804457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:00.804426 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:52:01.185042 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:01.185007 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:02.262615 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:02.262580 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:03.038401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:03.038355 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:03.758508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:03.758477 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:04.636885 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:04.635837 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:05.467636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:05.467599 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:06.409001 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:06.408948 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:07.278687 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.278650 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:07.754492 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.754457 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds"] Nov 02 09:52:07.759972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.759945 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.764765 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.764737 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmtm7\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-kube-api-access-xmtm7\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.764908 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.764772 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.764908 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.764798 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57315ea7-23ff-442b-8302-c909afc112cc-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.765001 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.764938 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.765066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.765050 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57315ea7-23ff-442b-8302-c909afc112cc-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.765118 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.765083 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.765182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.765153 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.765228 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.765217 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.765384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.765344 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.777756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.777725 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds"] Nov 02 09:52:07.866346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866311 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmtm7\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-kube-api-access-xmtm7\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866349 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866618 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866371 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57315ea7-23ff-442b-8302-c909afc112cc-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866618 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866453 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866618 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866563 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57315ea7-23ff-442b-8302-c909afc112cc-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866618 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866594 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866639 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866691 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866756 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.866845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866792 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.867082 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.866992 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.867216 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.867195 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.867409 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.867383 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57315ea7-23ff-442b-8302-c909afc112cc-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.867476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.867410 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.868933 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.868879 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.869266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.869246 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57315ea7-23ff-442b-8302-c909afc112cc-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.879534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.879502 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:07.879671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:07.879535 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmtm7\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-kube-api-access-xmtm7\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:08.068971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:08.068936 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:08.076746 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:08.076719 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:08.363604 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:08.363576 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds"] Nov 02 09:52:08.365275 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:52:08.365249 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57315ea7_23ff_442b_8302_c909afc112cc.slice/crio-cf293394f0b8be81a75c107954e22b7e9386ac7feb9b04215a845ad97dda5517 WatchSource:0}: Error finding container cf293394f0b8be81a75c107954e22b7e9386ac7feb9b04215a845ad97dda5517: Status 404 returned error can't find the container with id cf293394f0b8be81a75c107954e22b7e9386ac7feb9b04215a845ad97dda5517 Nov 02 09:52:08.367925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:08.367863 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:52:08.368008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:08.367962 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:52:08.368008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:08.367993 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:52:08.858657 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:08.858040 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:09.323038 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:09.322996 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" event={"ID":"57315ea7-23ff-442b-8302-c909afc112cc","Type":"ContainerStarted","Data":"ad925f30939d1b739816cbd573ffa2b2da6a2a87211f6db6afba83add7c0000b"} Nov 02 09:52:09.323226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:09.323106 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" event={"ID":"57315ea7-23ff-442b-8302-c909afc112cc","Type":"ContainerStarted","Data":"cf293394f0b8be81a75c107954e22b7e9386ac7feb9b04215a845ad97dda5517"} Nov 02 09:52:09.351708 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:09.351651 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" podStartSLOduration=2.351635607 podStartE2EDuration="2.351635607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:09.350271965 +0000 UTC m=+1008.190213958" watchObservedRunningTime="2026-11-02 09:52:09.351635607 +0000 UTC m=+1008.191577648" Nov 02 09:52:09.744470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:09.744384 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:10.077775 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:10.077740 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:10.079762 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:10.079725 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 09:52:10.079931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:10.079821 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 09:52:10.475008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:10.474868 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:11.016344 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:11.016308 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:11.078198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:11.077842 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 09:52:11.078198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:11.077912 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 09:52:11.235621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:11.235589 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:12.077570 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:12.077537 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 09:52:12.078005 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:12.077621 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 09:52:12.265916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:12.262830 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:12.623913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:12.623869 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:12.815915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:12.812445 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:13.077767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:13.077735 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 09:52:13.078205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:13.077789 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 09:52:13.167244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:13.167030 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:14.077424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:14.077378 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 09:52:14.077625 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:14.077453 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 09:52:14.166482 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:14.166449 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:14.955508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:14.955468 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:15.077660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:15.077619 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 09:52:15.077963 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:15.077928 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 09:52:15.316810 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:15.316766 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:15.512851 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:15.508353 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:15.772360 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:15.764464 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:16.298925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:16.298351 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:16.411534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:16.410238 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:16.411534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:16.411379 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:16.424955 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:16.420403 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:52:16.825913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:16.825398 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:17.020426 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:17.020391 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:17.526308 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:17.520980 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:18.296199 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:18.296166 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:19.183899 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:19.183852 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:19.960659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:19.960583 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:20.378109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:20.378070 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:20.560266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:20.560236 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:20.866497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:20.866446 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:21.578413 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:21.578376 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:21.779786 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:21.779753 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:22.565694 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:22.565660 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:22.868655 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:22.868554 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:52:23.361036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:23.355914 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:24.466511 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:24.466467 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:25.864448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:25.864412 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:26.766276 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:26.766216 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:26.859844 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:52:26.859810 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:52:27.521514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:27.521483 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:27.707784 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:27.707747 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:28.390419 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:28.390385 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:29.145916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:29.145859 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:29.847337 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:29.847303 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:30.542105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:30.542069 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:31.356944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:31.356910 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:32.206613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:32.206580 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:32.934872 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:32.934836 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:33.678946 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:33.678911 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:34.349320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:34.348766 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:35.031039 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:35.031003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:35.894240 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:35.894202 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:36.566802 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:36.566763 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:37.664338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:37.664296 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:37.677174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:37.677148 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b"] Nov 02 09:52:37.677455 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:37.677416 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" containerID="cri-o://3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214" gracePeriod=30 Nov 02 09:52:38.702403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:38.702307 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:40.609655 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:40.609618 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:41.484045 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.484007 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:41.848461 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.848421 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd"] Nov 02 09:52:41.852459 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.852439 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.867491 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.867463 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd"] Nov 02 09:52:41.903179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.903139 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.903506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.903479 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.903706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.903679 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.903815 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.903755 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7z4\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-kube-api-access-nw7z4\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.903874 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.903820 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.903944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.903911 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.904006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.903976 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.904105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.904088 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa532a-03d1-4d48-92c1-3057fab0f606-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:41.904163 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:41.904143 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.005689 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.005651 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.005872 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.005760 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa532a-03d1-4d48-92c1-3057fab0f606-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.005872 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.005805 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.005872 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.005849 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.005924 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.005982 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.006039 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7z4\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-kube-api-access-nw7z4\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.006073 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.006129 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.006320 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006483 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.006345 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.006565 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.006604 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.006720 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.006689 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa532a-03d1-4d48-92c1-3057fab0f606-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.008588 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.008561 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.008588 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.008568 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.018442 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.018415 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.019277 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.019251 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7z4\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-kube-api-access-nw7z4\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.169172 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.169082 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:42.273846 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.273075 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:42.487432 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.487399 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd"] Nov 02 09:52:42.488223 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:52:42.488191 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5aa532a_03d1_4d48_92c1_3057fab0f606.slice/crio-9af96f865d820a19213f485dbffad69c8153fe9c6399645e11012976c4c34cea WatchSource:0}: Error finding container 9af96f865d820a19213f485dbffad69c8153fe9c6399645e11012976c4c34cea: Status 404 returned error can't find the container with id 9af96f865d820a19213f485dbffad69c8153fe9c6399645e11012976c4c34cea Nov 02 09:52:42.491132 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.491091 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:52:42.491231 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.491178 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:52:42.491275 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.491240 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:52:42.739599 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.739500 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" event={"ID":"d5aa532a-03d1-4d48-92c1-3057fab0f606","Type":"ContainerStarted","Data":"69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd"} Nov 02 09:52:42.739599 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.739577 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" event={"ID":"d5aa532a-03d1-4d48-92c1-3057fab0f606","Type":"ContainerStarted","Data":"9af96f865d820a19213f485dbffad69c8153fe9c6399645e11012976c4c34cea"} Nov 02 09:52:42.768573 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.768510 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" podStartSLOduration=1.768492353 podStartE2EDuration="1.768492353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:42.764569667 +0000 UTC m=+1041.604511667" watchObservedRunningTime="2026-11-02 09:52:42.768492353 +0000 UTC m=+1041.608434350" Nov 02 09:52:42.950807 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:42.950777 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:52:43.126833 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.126791 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-kube-api-access-zcpng\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-kube-api-access-zcpng\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127090 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.126859 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-data\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127090 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127006 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-certs\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127090 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127045 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-podinfo\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127136 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-envoy\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127152 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-data" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:52:43.127270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127176 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-socket\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127278 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-credential-socket\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127289 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-certs" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:52:43.127425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127375 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-token\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-token\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127407 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istiod-ca-cert\") pod \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\" (UID: \"4bc586fd-8e80-41bf-8d00-5f83e36107f7\") " Nov 02 09:52:43.127629 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127504 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-credential-socket" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:52:43.127629 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127523 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-socket" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:52:43.127883 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127851 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istiod-ca-cert" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:52:43.128008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.127985 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.128079 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.128009 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.128079 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.128024 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.128079 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.128042 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.128079 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.128055 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.129751 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.129721 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-podinfo" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:52:43.129751 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.129727 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-envoy" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:52:43.129751 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.129740 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-kube-api-access-zcpng" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "kube-api-access-zcpng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:52:43.130067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.129928 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-token" pod "4bc586fd-8e80-41bf-8d00-5f83e36107f7" (UID: "4bc586fd-8e80-41bf-8d00-5f83e36107f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:52:43.169806 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.169775 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:43.171320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.171294 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 09:52:43.171402 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.171354 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 09:52:43.229469 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.229435 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.229469 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.229468 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcpng\" (UniqueName: \"kubernetes.io/projected/4bc586fd-8e80-41bf-8d00-5f83e36107f7-kube-api-access-zcpng\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.229671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.229479 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.229671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.229488 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc586fd-8e80-41bf-8d00-5f83e36107f7-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:52:43.381102 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.381047 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:43.750197 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.750096 4467 generic.go:356] "Generic (PLEG): container finished" podID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerID="3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214" exitCode=0 Nov 02 09:52:43.750197 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.750150 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" Nov 02 09:52:43.750197 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.750177 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" event={"ID":"4bc586fd-8e80-41bf-8d00-5f83e36107f7","Type":"ContainerDied","Data":"3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214"} Nov 02 09:52:43.750575 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.750251 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b" event={"ID":"4bc586fd-8e80-41bf-8d00-5f83e36107f7","Type":"ContainerDied","Data":"9ec5d2e5b7dbfa79e01e79ba09c5409ecc5a64bfb0be1507060794238d284da2"} Nov 02 09:52:43.750575 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.750282 4467 scope.go:122] "RemoveContainer" containerID="3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214" Nov 02 09:52:43.761714 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.761698 4467 scope.go:122] "RemoveContainer" containerID="3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214" Nov 02 09:52:43.762036 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:52:43.762017 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214\": container with ID starting with 3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214 not found: ID does not exist" containerID="3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214" Nov 02 09:52:43.762095 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.762046 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214"} err="failed to get container status \"3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214\": rpc error: code = NotFound desc = could not find container \"3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214\": container with ID starting with 3f0d4e7f562f79e7657b0bd56e0553007d23fde7988fb7654d60fb4c5f262214 not found: ID does not exist" Nov 02 09:52:43.770612 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.770589 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b"] Nov 02 09:52:43.775029 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:43.775007 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6pk2b"] Nov 02 09:52:44.175914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:44.175354 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 09:52:44.175914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:44.175422 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 09:52:44.702026 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:44.701880 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:45.169997 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:45.169963 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 09:52:45.170172 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:45.170019 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 09:52:45.332332 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:45.332297 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" path="/var/lib/kubelet/pods/4bc586fd-8e80-41bf-8d00-5f83e36107f7/volumes" Nov 02 09:52:45.512966 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:45.512843 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:45.689383 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:45.689348 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:46.169627 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:46.169594 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 09:52:46.169795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:46.169646 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 09:52:46.372248 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:46.372214 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:47.169478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:47.169432 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 09:52:47.169665 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:47.169514 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 09:52:47.481365 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:47.481246 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:48.170446 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:48.170411 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 09:52:48.170627 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:48.170473 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 09:52:48.263261 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:48.263216 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:49.068691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:49.068652 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:49.338582 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:49.338468 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:49.339125 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:49.339100 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:49.340453 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:49.340435 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:52:49.841480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:49.841445 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:50.363469 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:50.363433 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:52:50.610224 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:50.610187 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:51.490921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:51.490846 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:52.715651 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:52.715614 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:54.146683 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:54.146652 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:55.098308 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:55.098267 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:56.145108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:56.145069 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:56.876842 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:52:56.876807 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:52:56.913970 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:56.913932 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:57.110918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:57.108460 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:57.802006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:57.801965 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:58.507730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:58.507697 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:59.248379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:59.248351 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:59.783690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:59.783660 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:52:59.969813 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:52:59.969768 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:00.170998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:00.170196 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:01.745953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:01.737508 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:02.010977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:02.004993 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:53:02.010977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:02.005070 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:53:02.246820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:02.246780 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:02.444649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:02.444612 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:02.834539 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:02.816262 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:03.035956 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:03.035922 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:03.348591 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:03.348560 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:04.191465 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:04.191419 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:04.886622 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:04.886577 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:05.069048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:05.069010 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:05.506213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:05.506177 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:05.687927 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:05.687483 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:06.566745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:06.566702 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:06.861830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:06.861713 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:07.063456 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:07.063413 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:07.778477 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:07.778444 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:07.965373 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:07.965334 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:08.783681 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:08.783641 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:09.568839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:09.568805 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:10.435751 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:10.435715 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:11.053098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:11.053070 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:11.240711 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:11.240681 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:11.860067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:11.860034 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:12.282598 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.282499 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:53:12.515538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.515502 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh"] Nov 02 09:53:12.516637 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.516607 4467 cpu_manager.go:414] "containerMap: removing container" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" Nov 02 09:53:12.516794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.516780 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" Nov 02 09:53:12.517066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.517050 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc586fd-8e80-41bf-8d00-5f83e36107f7" containerName="istio-proxy" Nov 02 09:53:12.531563 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.531539 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.535296 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.535216 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh"] Nov 02 09:53:12.538977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.538955 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-87e02313-serving-cert\"" Nov 02 09:53:12.589907 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.589843 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-metrics-certs\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.589907 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.589907 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-maas-api-tls\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.590157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.589969 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c248e5-682b-49ea-8dde-354a1d542474-openshift-service-ca\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.590157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.590087 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbcc\" (UniqueName: \"kubernetes.io/projected/a9c248e5-682b-49ea-8dde-354a1d542474-kube-api-access-7qbcc\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.654478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.654435 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:12.692062 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.692019 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-metrics-certs\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.692251 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.692070 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-maas-api-tls\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.692251 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.692133 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c248e5-682b-49ea-8dde-354a1d542474-openshift-service-ca\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.692434 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.692256 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbcc\" (UniqueName: \"kubernetes.io/projected/a9c248e5-682b-49ea-8dde-354a1d542474-kube-api-access-7qbcc\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.692909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.692854 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c248e5-682b-49ea-8dde-354a1d542474-openshift-service-ca\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.695286 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.695263 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-metrics-certs\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.695488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.695461 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-maas-api-tls\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.706291 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.706221 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbcc\" (UniqueName: \"kubernetes.io/projected/a9c248e5-682b-49ea-8dde-354a1d542474-kube-api-access-7qbcc\") pod \"maas-api-e2e-ait-87e02313-856c7f468-tqzxh\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:12.844823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.844785 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:12.852758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:12.852731 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:13.638248 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:13.618216 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh"] Nov 02 09:53:13.659419 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:53:13.658192 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c248e5_682b_49ea_8dde_354a1d542474.slice/crio-3dd80b137d572ba2b47f4d4ce6ed6a38338680e09efc640644426c283acedbe6 WatchSource:0}: Error finding container 3dd80b137d572ba2b47f4d4ce6ed6a38338680e09efc640644426c283acedbe6: Status 404 returned error can't find the container with id 3dd80b137d572ba2b47f4d4ce6ed6a38338680e09efc640644426c283acedbe6 Nov 02 09:53:13.968966 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:13.968923 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:14.170107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:14.166874 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:14.178938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:14.178877 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" event={"ID":"a9c248e5-682b-49ea-8dde-354a1d542474","Type":"ContainerStarted","Data":"3dd80b137d572ba2b47f4d4ce6ed6a38338680e09efc640644426c283acedbe6"} Nov 02 09:53:14.392917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:14.390482 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:15.237435 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:15.237381 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" event={"ID":"a9c248e5-682b-49ea-8dde-354a1d542474","Type":"ContainerStarted","Data":"44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d"} Nov 02 09:53:15.239166 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:15.239121 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:15.266245 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:15.266179 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" podStartSLOduration=2.68152815 podStartE2EDuration="3.266141891s" totalImagesPullingTime="584.613741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:15.261376845 +0000 UTC m=+1074.101318855" watchObservedRunningTime="2026-11-02 09:53:15.266141891 +0000 UTC m=+1074.106083890" Nov 02 09:53:15.848820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:15.848786 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:16.763805 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:16.763753 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:17.636949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:17.630823 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:18.340639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:18.340602 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:19.037558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:19.037515 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:19.751498 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:19.751258 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:20.749201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:20.749162 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:21.756935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:21.756878 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:22.263063 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:22.263032 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:22.469159 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:22.469108 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:23.159989 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:23.159957 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:24.047813 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:24.047765 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:24.759391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:24.759355 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:25.466450 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:25.466407 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:26.306554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:26.305698 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:26.879397 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:53:26.879355 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:53:27.617678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:27.617630 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:28.988737 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:28.988701 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:29.966720 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:29.966680 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:30.404429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:30.404392 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:30.619992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:30.619965 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:30.804542 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:30.804433 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:31.559967 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:31.559932 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:32.246189 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:32.246153 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:32.952497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:32.952457 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:33.194365 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:33.194328 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:33.373483 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:33.373443 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:33.700935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:33.700829 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:34.551767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:34.551721 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:35.258797 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:35.258763 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:36.178362 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:36.178321 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:36.582597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:36.582561 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:53:36.957752 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:36.957511 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:37.843821 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:37.843745 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:38.535592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:38.535553 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:39.239078 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:39.239029 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:39.923339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:39.923302 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:40.767668 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:40.767630 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:41.457519 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:41.457487 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:42.156041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:42.156006 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:42.851765 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:42.851730 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:43.549666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:43.549627 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:44.246269 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:44.246238 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:45.067832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:45.067798 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:46.421969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:46.421937 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:46.620916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:46.619370 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:47.133948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:47.122373 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:48.029521 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:48.029470 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:48.284976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:48.269207 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:49.126031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:49.109513 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:50.173671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:50.173634 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:50.352072 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:50.352034 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:50.535817 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:50.535635 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:51.015311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.015279 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh"] Nov 02 09:53:51.015870 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.015838 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" podUID="a9c248e5-682b-49ea-8dde-354a1d542474" containerName="maas-api" containerID="cri-o://44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d" gracePeriod=30 Nov 02 09:53:51.160965 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.160922 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:51.390227 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.388318 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:51.426433 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.426401 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:51.450291 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.450262 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c248e5-682b-49ea-8dde-354a1d542474-kube-api-access-7qbcc\" (UniqueName: \"kubernetes.io/projected/a9c248e5-682b-49ea-8dde-354a1d542474-kube-api-access-7qbcc\") pod \"a9c248e5-682b-49ea-8dde-354a1d542474\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " Nov 02 09:53:51.450444 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.450383 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-maas-api-tls\") pod \"a9c248e5-682b-49ea-8dde-354a1d542474\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " Nov 02 09:53:51.450444 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.450425 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c248e5-682b-49ea-8dde-354a1d542474-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c248e5-682b-49ea-8dde-354a1d542474-openshift-service-ca\") pod \"a9c248e5-682b-49ea-8dde-354a1d542474\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " Nov 02 09:53:51.450591 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.450460 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-metrics-certs\") pod \"a9c248e5-682b-49ea-8dde-354a1d542474\" (UID: \"a9c248e5-682b-49ea-8dde-354a1d542474\") " Nov 02 09:53:51.451071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.451026 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c248e5-682b-49ea-8dde-354a1d542474-openshift-service-ca" pod "a9c248e5-682b-49ea-8dde-354a1d542474" (UID: "a9c248e5-682b-49ea-8dde-354a1d542474"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:53:51.454660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.454622 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c248e5-682b-49ea-8dde-354a1d542474-kube-api-access-7qbcc" pod "a9c248e5-682b-49ea-8dde-354a1d542474" (UID: "a9c248e5-682b-49ea-8dde-354a1d542474"). InnerVolumeSpecName "kube-api-access-7qbcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:53:51.455305 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.455260 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-maas-api-tls" pod "a9c248e5-682b-49ea-8dde-354a1d542474" (UID: "a9c248e5-682b-49ea-8dde-354a1d542474"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:53:51.456326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.456301 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-metrics-certs" pod "a9c248e5-682b-49ea-8dde-354a1d542474" (UID: "a9c248e5-682b-49ea-8dde-354a1d542474"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:53:51.553869 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.553837 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:53:51.553869 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.553865 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qbcc\" (UniqueName: \"kubernetes.io/projected/a9c248e5-682b-49ea-8dde-354a1d542474-kube-api-access-7qbcc\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:53:51.553869 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.553876 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c248e5-682b-49ea-8dde-354a1d542474-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:53:51.554134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.553885 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c248e5-682b-49ea-8dde-354a1d542474-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:53:51.565542 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.565513 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:51.583054 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.583018 4467 generic.go:356] "Generic (PLEG): container finished" podID="a9c248e5-682b-49ea-8dde-354a1d542474" containerID="44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d" exitCode=0 Nov 02 09:53:51.583222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.583077 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" Nov 02 09:53:51.583222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.583107 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" event={"ID":"a9c248e5-682b-49ea-8dde-354a1d542474","Type":"ContainerDied","Data":"44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d"} Nov 02 09:53:51.583222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.583161 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh" event={"ID":"a9c248e5-682b-49ea-8dde-354a1d542474","Type":"ContainerDied","Data":"3dd80b137d572ba2b47f4d4ce6ed6a38338680e09efc640644426c283acedbe6"} Nov 02 09:53:51.583222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.583183 4467 scope.go:122] "RemoveContainer" containerID="44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d" Nov 02 09:53:51.634910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.619498 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh"] Nov 02 09:53:51.634910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.627132 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87e02313-856c7f468-tqzxh"] Nov 02 09:53:51.654112 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.650320 4467 scope.go:122] "RemoveContainer" containerID="44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d" Nov 02 09:53:51.668001 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:53:51.667280 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d\": container with ID starting with 44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d not found: ID does not exist" containerID="44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d" Nov 02 09:53:51.668001 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.667324 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d"} err="failed to get container status \"44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d\": rpc error: code = NotFound desc = could not find container \"44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d\": container with ID starting with 44829bc5f1a6739da4a7de31480f4358ef0087ab49fca63ee4dcbdf7afdd061d not found: ID does not exist" Nov 02 09:53:51.748878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:51.748829 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:52.177992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:52.177954 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:52.803065 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:52.803025 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:52.980024 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:52.979990 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:53.180973 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:53.180941 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:53.407493 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:53.407457 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c248e5-682b-49ea-8dde-354a1d542474" path="/var/lib/kubelet/pods/a9c248e5-682b-49ea-8dde-354a1d542474/volumes" Nov 02 09:53:53.441998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:53.441377 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:53.521932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:53.519313 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd"] Nov 02 09:53:53.521932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:53.519542 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" podUID="2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" containerName="maas-api" containerID="cri-o://b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440" gracePeriod=30 Nov 02 09:53:53.623706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:53.623654 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:53.871279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:53.869568 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:53.937029 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:53.937001 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:53:54.054125 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.054013 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:54.094335 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.094307 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-openshift-service-ca\") pod \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " Nov 02 09:53:54.094523 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.094369 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-kube-api-access-qx9xg\" (UniqueName: \"kubernetes.io/projected/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-kube-api-access-qx9xg\") pod \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " Nov 02 09:53:54.094523 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.094509 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-metrics-certs\") pod \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " Nov 02 09:53:54.094646 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.094599 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-maas-api-tls\") pod \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\" (UID: \"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de\") " Nov 02 09:53:54.094710 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.094662 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-openshift-service-ca" pod "2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" (UID: "2bb7095c-ce6c-4164-a4d5-d80b62ccc9de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:53:54.101350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.096209 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:53:54.101350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.101248 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-maas-api-tls" pod "2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" (UID: "2bb7095c-ce6c-4164-a4d5-d80b62ccc9de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:53:54.101350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.101291 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-metrics-certs" pod "2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" (UID: "2bb7095c-ce6c-4164-a4d5-d80b62ccc9de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:53:54.102400 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.102369 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-kube-api-access-qx9xg" pod "2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" (UID: "2bb7095c-ce6c-4164-a4d5-d80b62ccc9de"). InnerVolumeSpecName "kube-api-access-qx9xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:53:54.197621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.197586 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:53:54.197621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.197617 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9xg\" (UniqueName: \"kubernetes.io/projected/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-kube-api-access-qx9xg\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:53:54.197621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.197630 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:53:54.307581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.307542 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:54.501648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.500285 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:54.647971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.647283 4467 generic.go:356] "Generic (PLEG): container finished" podID="2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" containerID="b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440" exitCode=0 Nov 02 09:53:54.647971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.647437 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" event={"ID":"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de","Type":"ContainerDied","Data":"b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440"} Nov 02 09:53:54.647971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.647490 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" event={"ID":"2bb7095c-ce6c-4164-a4d5-d80b62ccc9de","Type":"ContainerDied","Data":"2d439a57a4cf834c36341f45077ca5ef2c172901f34b91d0cae2173edb2b8e20"} Nov 02 09:53:54.647971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.647520 4467 scope.go:122] "RemoveContainer" containerID="b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440" Nov 02 09:53:54.647971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.647668 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd" Nov 02 09:53:54.698046 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.687236 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:54.734634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.734252 4467 scope.go:122] "RemoveContainer" containerID="b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440" Nov 02 09:53:54.735987 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:53:54.735846 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440\": container with ID starting with b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440 not found: ID does not exist" containerID="b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440" Nov 02 09:53:54.735987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.735913 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440"} err="failed to get container status \"b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440\": rpc error: code = NotFound desc = could not find container \"b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440\": container with ID starting with b44c8a7fe2d9c2b75c316170a4717cb8180d6ef3d810ca306267bcc27fc2e440 not found: ID does not exist" Nov 02 09:53:54.766911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.763046 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd"] Nov 02 09:53:54.768415 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:54.768319 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-x8twd"] Nov 02 09:53:55.332730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:55.332695 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" path="/var/lib/kubelet/pods/2bb7095c-ce6c-4164-a4d5-d80b62ccc9de/volumes" Nov 02 09:53:55.766472 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:55.765373 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:56.318097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:56.318062 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:56.509218 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:56.509166 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:56.759236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:56.759147 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:56.854720 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:53:56.854688 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:53:57.461965 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:57.461858 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:57.656059 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:57.655986 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:58.347374 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:58.347340 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:58.547224 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:58.547193 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:59.211245 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.211181 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:59.794458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.793143 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g"] Nov 02 09:53:59.794458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.794013 4467 cpu_manager.go:414] "containerMap: removing container" podUID="a9c248e5-682b-49ea-8dde-354a1d542474" containerName="maas-api" Nov 02 09:53:59.794458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.794031 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c248e5-682b-49ea-8dde-354a1d542474" containerName="maas-api" Nov 02 09:53:59.794458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.794055 4467 cpu_manager.go:414] "containerMap: removing container" podUID="2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" containerName="maas-api" Nov 02 09:53:59.794458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.794064 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" containerName="maas-api" Nov 02 09:53:59.794458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.794191 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c248e5-682b-49ea-8dde-354a1d542474" containerName="maas-api" Nov 02 09:53:59.794458 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.794204 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb7095c-ce6c-4164-a4d5-d80b62ccc9de" containerName="maas-api" Nov 02 09:53:59.807706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.807681 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:53:59.809351 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.809287 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g"] Nov 02 09:53:59.816804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.816782 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-ced1b7-serving-cert\"" Nov 02 09:53:59.947254 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.947223 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:53:59.984375 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.984339 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mncsg\" (UniqueName: \"kubernetes.io/projected/01a4b738-51a6-448f-a225-3e25ce2c3378-kube-api-access-mncsg\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:53:59.984375 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.984392 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a4b738-51a6-448f-a225-3e25ce2c3378-openshift-service-ca\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:53:59.984671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.984469 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-maas-api-tls\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:53:59.984671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:53:59.984594 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-metrics-certs\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.086256 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.086217 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-maas-api-tls\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.086443 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.086344 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-metrics-certs\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.086497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.086456 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mncsg\" (UniqueName: \"kubernetes.io/projected/01a4b738-51a6-448f-a225-3e25ce2c3378-kube-api-access-mncsg\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.086548 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.086507 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a4b738-51a6-448f-a225-3e25ce2c3378-openshift-service-ca\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.087203 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.087179 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a4b738-51a6-448f-a225-3e25ce2c3378-openshift-service-ca\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.088945 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.088921 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-maas-api-tls\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.089349 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.089326 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-metrics-certs\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.100537 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.100506 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mncsg\" (UniqueName: \"kubernetes.io/projected/01a4b738-51a6-448f-a225-3e25ce2c3378-kube-api-access-mncsg\") pod \"maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.129427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.129398 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:00.158589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.158557 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:00.417845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.417762 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g"] Nov 02 09:54:00.596404 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.596365 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:54:00.699347 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.699267 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" event={"ID":"01a4b738-51a6-448f-a225-3e25ce2c3378","Type":"ContainerStarted","Data":"4c75eca72f519caa1e9c5a93e7c25c22a33956c499ccbaccb9fb6b747af7d44d"} Nov 02 09:54:00.971116 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:00.969485 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:01.754916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:01.754013 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" event={"ID":"01a4b738-51a6-448f-a225-3e25ce2c3378","Type":"ContainerStarted","Data":"8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd"} Nov 02 09:54:01.755416 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:01.755121 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:01.792913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:01.781485 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" podStartSLOduration=2.250821005 podStartE2EDuration="2.781463335s" totalImagesPullingTime="530.64233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:01.778654163 +0000 UTC m=+1120.618596161" watchObservedRunningTime="2026-11-02 09:54:01.781463335 +0000 UTC m=+1120.621405331" Nov 02 09:54:02.276510 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:02.276471 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:02.473289 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:02.473224 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:02.702941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:02.702872 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:02.866716 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:02.866679 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:54:02.867359 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:02.867323 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:54:03.150858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:03.150818 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:03.350232 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:03.350191 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:03.707697 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:03.707658 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:03.906127 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:03.906088 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:04.715083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.715046 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4"] Nov 02 09:54:04.720632 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.720607 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.729386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.729355 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4"] Nov 02 09:54:04.731157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.731125 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-13c6a8db-serving-cert\"" Nov 02 09:54:04.783772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.783732 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.784003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.783885 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b463bab2-427d-474c-9a6c-4809627a8fe2-openshift-service-ca\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.784003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.783980 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-metrics-certs\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.784136 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.784011 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z56xt\" (UniqueName: \"kubernetes.io/projected/b463bab2-427d-474c-9a6c-4809627a8fe2-kube-api-access-z56xt\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.877669 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.877634 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:04.899317 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.899278 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.899855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.899829 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b463bab2-427d-474c-9a6c-4809627a8fe2-openshift-service-ca\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.900122 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.900021 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-metrics-certs\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.900286 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.900265 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z56xt\" (UniqueName: \"kubernetes.io/projected/b463bab2-427d-474c-9a6c-4809627a8fe2-kube-api-access-z56xt\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.900818 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:54:04.899708 4467 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-serving-cert: secret "maas-api-e2e-derive-13c6a8db-serving-cert" not found Nov 02 09:54:04.901086 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:54:04.901054 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls podName:b463bab2-427d-474c-9a6c-4809627a8fe2 nodeName:}" failed. No retries permitted until 2026-11-02 09:54:05.401028802 +0000 UTC m=+1124.240970796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls") pod "maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" (UID: "b463bab2-427d-474c-9a6c-4809627a8fe2") : secret "maas-api-e2e-derive-13c6a8db-serving-cert" not found Nov 02 09:54:04.906196 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.906161 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b463bab2-427d-474c-9a6c-4809627a8fe2-openshift-service-ca\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.915355 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.913802 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-metrics-certs\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:04.927362 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:04.927331 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z56xt\" (UniqueName: \"kubernetes.io/projected/b463bab2-427d-474c-9a6c-4809627a8fe2-kube-api-access-z56xt\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:05.065921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:05.065863 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:05.246493 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:05.246452 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:05.421970 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:05.416750 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:05.425921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:05.422421 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls\") pod \"maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:05.655136 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:05.655105 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:05.797934 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:05.791133 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:05.966884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:05.966857 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4"] Nov 02 09:54:05.969790 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:54:05.969762 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb463bab2_427d_474c_9a6c_4809627a8fe2.slice/crio-37ed91fc687c2a02d80caacec1bc56e3b90bc725b88b36e3d1c85189eeee4fd3 WatchSource:0}: Error finding container 37ed91fc687c2a02d80caacec1bc56e3b90bc725b88b36e3d1c85189eeee4fd3: Status 404 returned error can't find the container with id 37ed91fc687c2a02d80caacec1bc56e3b90bc725b88b36e3d1c85189eeee4fd3 Nov 02 09:54:06.471424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:06.471374 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:06.820092 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:06.820055 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" event={"ID":"b463bab2-427d-474c-9a6c-4809627a8fe2","Type":"ContainerStarted","Data":"7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20"} Nov 02 09:54:06.820336 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:06.820120 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" event={"ID":"b463bab2-427d-474c-9a6c-4809627a8fe2","Type":"ContainerStarted","Data":"37ed91fc687c2a02d80caacec1bc56e3b90bc725b88b36e3d1c85189eeee4fd3"} Nov 02 09:54:06.820336 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:06.820248 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:06.844956 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:06.844296 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" podStartSLOduration=2.340198222 podStartE2EDuration="2.844277543s" totalImagesPullingTime="504.079321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:06.842061029 +0000 UTC m=+1125.682003025" watchObservedRunningTime="2026-11-02 09:54:06.844277543 +0000 UTC m=+1125.684219542" Nov 02 09:54:07.377173 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:07.377139 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:07.997048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:07.996999 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf"] Nov 02 09:54:08.002269 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.002243 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.057752 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.050850 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf"] Nov 02 09:54:08.059787 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.059748 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.059977 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.059928 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.060100 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.060070 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.060169 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.060110 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sch6q\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-kube-api-access-sch6q\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.060169 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.060155 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.060262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.060194 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.060262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.060250 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.060328 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.060284 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.060328 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.060308 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aed7e46-1726-4cbc-bac5-45a69cbce271-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.063390 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.063323 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:08.161239 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161202 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161244 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sch6q\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-kube-api-access-sch6q\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161307 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161351 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161591 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161506 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161563 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161616 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aed7e46-1726-4cbc-bac5-45a69cbce271-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161749 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161713 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161801 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161772 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161854 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161808 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.161969 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.161940 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.162105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.162083 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.162205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.162180 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.162408 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.162391 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aed7e46-1726-4cbc-bac5-45a69cbce271-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.164291 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.164273 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.164541 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.164522 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.174205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.174178 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.174340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.174189 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sch6q\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-kube-api-access-sch6q\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.351529 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.351497 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:08.637931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.637885 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf"] Nov 02 09:54:08.639391 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:54:08.639362 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aed7e46_1726_4cbc_bac5_45a69cbce271.slice/crio-71c407c8a8cca95430c67d70940e8d86530925275d7d9bcf83f4e689c7452b65 WatchSource:0}: Error finding container 71c407c8a8cca95430c67d70940e8d86530925275d7d9bcf83f4e689c7452b65: Status 404 returned error can't find the container with id 71c407c8a8cca95430c67d70940e8d86530925275d7d9bcf83f4e689c7452b65 Nov 02 09:54:08.642027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.641991 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:08.642129 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.642072 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:08.642129 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.642116 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:08.821506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.821480 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:54:08.861957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.861877 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" event={"ID":"7aed7e46-1726-4cbc-bac5-45a69cbce271","Type":"ContainerStarted","Data":"1302bfd5f6eed8d071513c33c8fb1371210e3b4b611f1ba69b7fd02576be57a2"} Nov 02 09:54:08.862132 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.861965 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" event={"ID":"7aed7e46-1726-4cbc-bac5-45a69cbce271","Type":"ContainerStarted","Data":"71c407c8a8cca95430c67d70940e8d86530925275d7d9bcf83f4e689c7452b65"} Nov 02 09:54:08.868562 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.868527 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:08.896307 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:08.896172 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podStartSLOduration=1.896133544 podStartE2EDuration="1.896133544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:08.892247132 +0000 UTC m=+1127.732189124" watchObservedRunningTime="2026-11-02 09:54:08.896133544 +0000 UTC m=+1127.736075596" Nov 02 09:54:09.352405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:09.352369 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:09.354346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:09.354319 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 09:54:09.354476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:09.354368 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 09:54:09.715009 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:09.696747 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:10.353938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:10.352489 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 09:54:10.353938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:10.352585 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 09:54:10.702164 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:10.702050 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:11.132756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:11.132718 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:11.352435 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:11.352393 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 09:54:11.352635 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:11.352473 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 09:54:11.477972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:11.477731 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:11.657630 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:11.657593 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:12.043830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.043788 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5"] Nov 02 09:54:12.048951 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.048926 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078181 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078147 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078194 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078251 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6628631-e521-471e-ab13-311df7e83a02-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078300 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078527 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078362 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078527 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078433 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078527 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078470 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078629 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wnz\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-kube-api-access-p5wnz\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.078754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.078706 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6628631-e521-471e-ab13-311df7e83a02-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.162932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.154354 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5"] Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180240 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180284 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180334 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6628631-e521-471e-ab13-311df7e83a02-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180376 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180432 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180493 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180527 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180575 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wnz\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-kube-api-access-p5wnz\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.181066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.180636 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6628631-e521-471e-ab13-311df7e83a02-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.186928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.181560 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.187180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.183546 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.188022 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.187995 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6628631-e521-471e-ab13-311df7e83a02-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.231071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.225315 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.231071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.225588 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.235727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.235673 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6628631-e521-471e-ab13-311df7e83a02-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.237238 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.237201 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.237638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.237613 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.239612 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.239587 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wnz\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-kube-api-access-p5wnz\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.352512 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.352474 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 09:54:12.352697 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.352544 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 09:54:12.466504 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.466471 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:12.665788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.665578 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:12.831585 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.831556 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:12.839409 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:54:12.839371 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6628631_e521_471e_ab13_311df7e83a02.slice/crio-0231b514a326b8c626256a2c8655f6a9b6634d41f01ea468fc5e80aee3144938 WatchSource:0}: Error finding container 0231b514a326b8c626256a2c8655f6a9b6634d41f01ea468fc5e80aee3144938: Status 404 returned error can't find the container with id 0231b514a326b8c626256a2c8655f6a9b6634d41f01ea468fc5e80aee3144938 Nov 02 09:54:12.840270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.840247 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5"] Nov 02 09:54:12.842243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.842211 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:12.842343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.842283 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:12.842343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.842331 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:12.958444 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.957805 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" event={"ID":"a6628631-e521-471e-ab13-311df7e83a02","Type":"ContainerStarted","Data":"a059a13084bbf9dbf00543641843091e48d68dbe851d6b4ee2272f2c95e6c939"} Nov 02 09:54:12.958444 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:12.957862 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" event={"ID":"a6628631-e521-471e-ab13-311df7e83a02","Type":"ContainerStarted","Data":"0231b514a326b8c626256a2c8655f6a9b6634d41f01ea468fc5e80aee3144938"} Nov 02 09:54:13.175962 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:13.175584 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:13.352472 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:13.352433 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 09:54:13.352728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:13.352695 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 09:54:13.467315 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:13.467247 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:13.468973 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:13.468939 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 09:54:13.469116 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:13.469006 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 09:54:13.815513 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:13.815467 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:14.352920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:14.352862 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 09:54:14.353339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:14.352928 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 09:54:14.467819 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:14.467784 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 09:54:14.468003 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:14.467841 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 09:54:14.856191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:14.826724 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:15.352769 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.352733 4467 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 09:54:15.353027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.352781 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 09:54:15.447439 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.447401 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:15.471961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.471885 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 09:54:15.472258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.472221 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 09:54:15.632594 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.632465 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podStartSLOduration=3.632442737 podStartE2EDuration="3.632442737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:13.039622079 +0000 UTC m=+1131.879564075" watchObservedRunningTime="2026-11-02 09:54:15.632442737 +0000 UTC m=+1134.472384733" Nov 02 09:54:15.633066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.633041 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:54:15.773927 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.761198 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:15.955638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:15.955529 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:16.482051 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:16.482014 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 09:54:16.482672 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:16.482633 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 09:54:16.817915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:16.816727 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:16.819575 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:16.819548 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:16.832198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:16.832168 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:54:17.481076 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:17.481039 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 09:54:17.481360 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:17.481328 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 09:54:18.467858 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:18.467824 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 09:54:18.468339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:18.467882 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 09:54:18.731650 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:18.731551 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:19.468204 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:19.468172 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 09:54:19.468789 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:19.468707 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 09:54:20.378873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:20.378834 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:20.467723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:20.467683 4467 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 09:54:20.467943 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:20.467763 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 09:54:21.281231 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:21.281196 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:21.472218 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:21.472186 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:21.472472 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:21.472447 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:21.473266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:21.473246 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:54:22.264487 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:22.264388 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:23.177602 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:23.177565 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:24.065682 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:24.065647 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:24.745551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:24.745516 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:25.721533 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:25.721490 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:26.700254 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:26.700218 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:26.856111 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:54:26.856073 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:54:26.876279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:26.876232 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:27.848396 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:27.827250 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:28.047087 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:28.047014 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:28.371913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:28.371077 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:29.060437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:29.060397 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:29.856665 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:29.856617 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:30.510634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:30.510565 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:30.926301 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:30.926257 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:31.110244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:31.110182 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:31.656125 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:31.656069 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:32.364480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:32.364451 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:32.654841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:32.654740 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:33.478055 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.478017 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:33.598243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.598211 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b"] Nov 02 09:54:33.603652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.603630 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.618868 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.618828 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b"] Nov 02 09:54:33.690741 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.690702 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.690942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.690759 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.690942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.690851 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.690942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.690916 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.691097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.691008 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.691097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.691035 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.691097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.691058 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.691245 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.691227 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.691361 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.691344 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dzh\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-kube-api-access-q7dzh\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.792936 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.792816 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.792943 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793044 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dzh\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-kube-api-access-q7dzh\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793105 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793139 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793195 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793261 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793312 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793349 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.793569 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793542 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.794140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.793945 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.794140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.794102 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.794331 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.794309 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.794398 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.794372 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.795595 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.795574 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.796666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.796642 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.805427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.805398 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.805520 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.805442 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dzh\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-kube-api-access-q7dzh\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:33.924668 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:33.924631 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:34.215661 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:54:34.215632 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c69cf9_ca53_4857_8b3d_47bf9b3aa331.slice/crio-6083801eaaf30cde4f3d7056a5c2ac624bac2af7a09525ea23ba5fc181f818f5 WatchSource:0}: Error finding container 6083801eaaf30cde4f3d7056a5c2ac624bac2af7a09525ea23ba5fc181f818f5: Status 404 returned error can't find the container with id 6083801eaaf30cde4f3d7056a5c2ac624bac2af7a09525ea23ba5fc181f818f5 Nov 02 09:54:34.216104 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.216079 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b"] Nov 02 09:54:34.218601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.218571 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:34.218711 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.218656 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:34.218711 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.218698 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:54:34.368468 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.368424 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:34.649312 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.649274 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" event={"ID":"04c69cf9-ca53-4857-8b3d-47bf9b3aa331","Type":"ContainerStarted","Data":"af310a99beea0845348ff632c86ece1b1767d3a60793034d59ef65ea3cddd14b"} Nov 02 09:54:34.649824 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.649337 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" event={"ID":"04c69cf9-ca53-4857-8b3d-47bf9b3aa331","Type":"ContainerStarted","Data":"6083801eaaf30cde4f3d7056a5c2ac624bac2af7a09525ea23ba5fc181f818f5"} Nov 02 09:54:34.678026 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.677976 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" podStartSLOduration=1.6779622239999998 podStartE2EDuration="1.677962224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:34.674915488 +0000 UTC m=+1153.514857478" watchObservedRunningTime="2026-11-02 09:54:34.677962224 +0000 UTC m=+1153.517904218" Nov 02 09:54:34.925593 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:34.925499 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:35.134202 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:35.132788 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:35.317007 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:35.316973 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:35.504914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:35.504561 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:35.927293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:35.927257 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 09:54:35.927705 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:35.927313 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 09:54:36.007821 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:36.006364 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:36.267482 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:36.267391 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:36.925913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:36.924988 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 09:54:36.925913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:36.925045 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 09:54:36.926182 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:36.925912 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4"] Nov 02 09:54:36.926247 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:36.926165 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" podUID="b463bab2-427d-474c-9a6c-4809627a8fe2" containerName="maas-api" containerID="cri-o://7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20" gracePeriod=30 Nov 02 09:54:37.044487 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.044452 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:37.240151 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.234757 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:37.456007 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.454985 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:37.622129 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.622094 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:37.669546 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.669513 4467 generic.go:356] "Generic (PLEG): container finished" podID="b463bab2-427d-474c-9a6c-4809627a8fe2" containerID="7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20" exitCode=0 Nov 02 09:54:37.669814 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.669566 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" Nov 02 09:54:37.669961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.669583 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" event={"ID":"b463bab2-427d-474c-9a6c-4809627a8fe2","Type":"ContainerDied","Data":"7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20"} Nov 02 09:54:37.670030 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.670010 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4" event={"ID":"b463bab2-427d-474c-9a6c-4809627a8fe2","Type":"ContainerDied","Data":"37ed91fc687c2a02d80caacec1bc56e3b90bc725b88b36e3d1c85189eeee4fd3"} Nov 02 09:54:37.670081 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.670040 4467 scope.go:122] "RemoveContainer" containerID="7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20" Nov 02 09:54:37.681558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.681535 4467 scope.go:122] "RemoveContainer" containerID="7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20" Nov 02 09:54:37.681813 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:54:37.681784 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20\": container with ID starting with 7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20 not found: ID does not exist" containerID="7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20" Nov 02 09:54:37.681928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.681826 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20"} err="failed to get container status \"7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20\": rpc error: code = NotFound desc = could not find container \"7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20\": container with ID starting with 7bdc4e431e315fe511536cf9335fc5966b1604d2d2dfc47e34462964c9757d20 not found: ID does not exist" Nov 02 09:54:37.743767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.743685 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:37.756644 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.755316 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-metrics-certs\") pod \"b463bab2-427d-474c-9a6c-4809627a8fe2\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " Nov 02 09:54:37.756644 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.755359 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b463bab2-427d-474c-9a6c-4809627a8fe2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b463bab2-427d-474c-9a6c-4809627a8fe2-openshift-service-ca\") pod \"b463bab2-427d-474c-9a6c-4809627a8fe2\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " Nov 02 09:54:37.756644 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.755411 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls\") pod \"b463bab2-427d-474c-9a6c-4809627a8fe2\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " Nov 02 09:54:37.756644 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.755471 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b463bab2-427d-474c-9a6c-4809627a8fe2-kube-api-access-z56xt\" (UniqueName: \"kubernetes.io/projected/b463bab2-427d-474c-9a6c-4809627a8fe2-kube-api-access-z56xt\") pod \"b463bab2-427d-474c-9a6c-4809627a8fe2\" (UID: \"b463bab2-427d-474c-9a6c-4809627a8fe2\") " Nov 02 09:54:37.756644 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.756317 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b463bab2-427d-474c-9a6c-4809627a8fe2-openshift-service-ca" pod "b463bab2-427d-474c-9a6c-4809627a8fe2" (UID: "b463bab2-427d-474c-9a6c-4809627a8fe2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:54:37.760841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.760800 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-metrics-certs" pod "b463bab2-427d-474c-9a6c-4809627a8fe2" (UID: "b463bab2-427d-474c-9a6c-4809627a8fe2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:54:37.761225 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.761199 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls" pod "b463bab2-427d-474c-9a6c-4809627a8fe2" (UID: "b463bab2-427d-474c-9a6c-4809627a8fe2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:54:37.761387 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.761219 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b463bab2-427d-474c-9a6c-4809627a8fe2-kube-api-access-z56xt" pod "b463bab2-427d-474c-9a6c-4809627a8fe2" (UID: "b463bab2-427d-474c-9a6c-4809627a8fe2"). InnerVolumeSpecName "kube-api-access-z56xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:54:37.880914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.878471 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:54:37.880914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.878507 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b463bab2-427d-474c-9a6c-4809627a8fe2-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:54:37.880914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.878522 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b463bab2-427d-474c-9a6c-4809627a8fe2-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:54:37.880914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.878536 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z56xt\" (UniqueName: \"kubernetes.io/projected/b463bab2-427d-474c-9a6c-4809627a8fe2-kube-api-access-z56xt\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:54:37.950499 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.950291 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:37.950769 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.950731 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 09:54:37.950875 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:37.950803 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 09:54:38.072319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:38.072282 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4"] Nov 02 09:54:38.117602 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:38.117569 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-13c6a8db-8dd94bc4d-z5cr4"] Nov 02 09:54:38.719237 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:38.719202 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:38.928451 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:38.928417 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 09:54:38.928798 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:38.928766 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 09:54:38.970861 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:38.970755 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:39.242711 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:39.242582 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:39.342146 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:39.342111 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b463bab2-427d-474c-9a6c-4809627a8fe2" path="/var/lib/kubelet/pods/b463bab2-427d-474c-9a6c-4809627a8fe2/volumes" Nov 02 09:54:39.533987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:39.533031 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:39.925870 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:39.925830 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 09:54:39.926197 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:39.926167 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 09:54:40.074721 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:40.074694 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:40.254706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:40.254617 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:54:40.936071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:40.936033 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:40.937183 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:40.937165 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:40.938175 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:40.938160 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 09:54:41.328494 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:41.328458 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:42.598236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:42.597119 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:42.849653 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:42.849553 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:43.155884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:43.155755 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:43.491008 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:43.490913 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:44.009718 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:44.009687 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:44.222863 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:44.222828 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:44.427950 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:44.427917 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:45.361281 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:45.361246 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:45.552911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:45.552317 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:46.263636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:46.263604 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:47.763279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:47.763235 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:48.455655 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:48.455624 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:49.157035 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:49.155665 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:49.941793 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:49.941760 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:50.682048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.682011 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz"] Nov 02 09:54:50.683000 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.682968 4467 cpu_manager.go:414] "containerMap: removing container" podUID="b463bab2-427d-474c-9a6c-4809627a8fe2" containerName="maas-api" Nov 02 09:54:50.683000 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.682996 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b463bab2-427d-474c-9a6c-4809627a8fe2" containerName="maas-api" Nov 02 09:54:50.683200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.683142 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="b463bab2-427d-474c-9a6c-4809627a8fe2" containerName="maas-api" Nov 02 09:54:50.688695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.688668 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.709284 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.709252 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz"] Nov 02 09:54:50.709417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.709395 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a71884-serving-cert\"" Nov 02 09:54:50.812547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.812511 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-metrics-certs\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.812736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.812579 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8w7\" (UniqueName: \"kubernetes.io/projected/13d82ead-327a-420f-82a2-efe567b10bda-kube-api-access-4p8w7\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.812736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.812714 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d82ead-327a-420f-82a2-efe567b10bda-openshift-service-ca\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.812836 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.812769 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-maas-api-tls\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.841195 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.841159 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:50.915913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.915695 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-metrics-certs\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.915913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.915779 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8w7\" (UniqueName: \"kubernetes.io/projected/13d82ead-327a-420f-82a2-efe567b10bda-kube-api-access-4p8w7\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.915913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.915918 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d82ead-327a-420f-82a2-efe567b10bda-openshift-service-ca\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.916228 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.915989 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-maas-api-tls\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.939474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.939395 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d82ead-327a-420f-82a2-efe567b10bda-openshift-service-ca\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.941778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.941712 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-maas-api-tls\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.942497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.942472 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-metrics-certs\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:50.945242 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:50.945221 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8w7\" (UniqueName: \"kubernetes.io/projected/13d82ead-327a-420f-82a2-efe567b10bda-kube-api-access-4p8w7\") pod \"maas-api-e2e-api-a-a71884-758b75998b-z2fmz\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:51.027932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:51.023491 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:51.028351 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:51.028328 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:51.204751 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:51.204632 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:51.360351 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:51.360320 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz"] Nov 02 09:54:51.361712 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:54:51.361684 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d82ead_327a_420f_82a2_efe567b10bda.slice/crio-c75cef81ede910db92f59280091d67ba2498c089c88b29aa56f4da81815b83a1 WatchSource:0}: Error finding container c75cef81ede910db92f59280091d67ba2498c089c88b29aa56f4da81815b83a1: Status 404 returned error can't find the container with id c75cef81ede910db92f59280091d67ba2498c089c88b29aa56f4da81815b83a1 Nov 02 09:54:51.848149 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:51.848108 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" event={"ID":"13d82ead-327a-420f-82a2-efe567b10bda","Type":"ContainerStarted","Data":"c75cef81ede910db92f59280091d67ba2498c089c88b29aa56f4da81815b83a1"} Nov 02 09:54:51.858013 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:51.857961 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:52.560987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:52.560947 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:52.855965 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:52.855845 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" event={"ID":"13d82ead-327a-420f-82a2-efe567b10bda","Type":"ContainerStarted","Data":"4217a91eabe1c913bb1ce5156e600c3186d7112ce7a7fc27db6d45a0415d84eb"} Nov 02 09:54:52.856376 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:52.856057 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:52.876984 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:52.876934 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" podStartSLOduration=2.380495104 podStartE2EDuration="2.876921318s" totalImagesPullingTime="496.426214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:52.874853606 +0000 UTC m=+1171.714795613" watchObservedRunningTime="2026-11-02 09:54:52.876921318 +0000 UTC m=+1171.716863309" Nov 02 09:54:53.296731 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:53.296643 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:53.994528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:53.994483 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:54.815111 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:54.815080 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:55.470769 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:55.470709 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:56.158574 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:56.158533 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:56.867445 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:56.867406 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:56.896567 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:54:56.896526 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:54:57.556098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:57.556044 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:58.247326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:58.247274 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:58.868566 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:58.868539 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:54:58.954971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:58.954916 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:54:59.647045 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:54:59.647013 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:00.353066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:00.353031 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:01.046555 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:01.046480 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:01.597772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:01.597724 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:55:01.760911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:01.760858 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:02.760453 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:02.760418 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:03.431938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:03.431902 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:04.154988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.154957 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:04.269049 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.269013 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk"] Nov 02 09:55:04.273828 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.273805 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.279437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.279415 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9cbd331c-openshift-default-dockercfg-t4z5f\"" Nov 02 09:55:04.287131 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.287103 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk"] Nov 02 09:55:04.440712 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.440618 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.440712 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.440665 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.440940 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.440814 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.440940 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.440832 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hfpd\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-kube-api-access-5hfpd\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.440940 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.440860 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.441075 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.440966 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.441075 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.441025 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.441179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.441150 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.441233 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.441212 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.542961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.542919 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543004 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543022 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543094 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543112 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfpd\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-kube-api-access-5hfpd\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543141 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543200 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543283 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543296 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543331 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543600 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543393 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543658 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.543751 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.543708 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.544424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.544394 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.545623 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.545603 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.545935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.545903 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.555234 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.555211 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.555391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.555373 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hfpd\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-kube-api-access-5hfpd\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.590876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.590843 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:04.883951 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.883923 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk"] Nov 02 09:55:04.885531 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:55:04.885500 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39ac5fb_6f24_453e_99ec_a0bdf072b4d1.slice/crio-910743d761d3c237adef81494b2ac064333c7cca5e93af514a65d204ecddf986 WatchSource:0}: Error finding container 910743d761d3c237adef81494b2ac064333c7cca5e93af514a65d204ecddf986: Status 404 returned error can't find the container with id 910743d761d3c237adef81494b2ac064333c7cca5e93af514a65d204ecddf986 Nov 02 09:55:04.888353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.888320 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:55:04.888456 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.888387 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:55:04.888456 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.888415 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:55:04.940165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.940076 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" event={"ID":"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1","Type":"ContainerStarted","Data":"910743d761d3c237adef81494b2ac064333c7cca5e93af514a65d204ecddf986"} Nov 02 09:55:04.947180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:04.947145 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:05.780521 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:05.780485 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:05.950638 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:05.950597 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" event={"ID":"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1","Type":"ContainerStarted","Data":"d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81"} Nov 02 09:55:05.976704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:05.976646 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" podStartSLOduration=1.976630653 podStartE2EDuration="1.976630653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:05.974126296 +0000 UTC m=+1184.814068294" watchObservedRunningTime="2026-11-02 09:55:05.976630653 +0000 UTC m=+1184.816572647" Nov 02 09:55:06.613912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:06.605255 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:06.613912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:06.612968 4467 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 09:55:06.613912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:06.613025 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 09:55:06.837650 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:06.837512 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:07.603930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:07.603334 4467 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 09:55:07.603930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:07.603396 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 09:55:07.863909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:07.863806 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:08.602909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:08.595418 4467 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 09:55:08.602909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:08.595497 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 09:55:08.651826 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:08.651429 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:08.958467 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:08.958367 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:09.562534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:09.562498 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:09.608882 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:09.608850 4467 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 09:55:09.609181 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:09.609154 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 09:55:09.744999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:09.744967 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:10.086972 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:10.086835 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:10.532973 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:10.530412 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:10.607154 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:10.607036 4467 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 09:55:10.607154 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:10.607091 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 09:55:10.764397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:10.764359 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:11.194213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:11.194179 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:11.707081 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:11.707046 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:11.707493 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:11.707465 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:11.708539 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:11.708411 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:11.760616 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:11.760573 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:12.545619 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:12.545552 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:12.806334 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:12.806224 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:13.013384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:13.013327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:13.650939 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:13.650870 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:13.836680 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:13.836647 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:14.044704 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:14.044674 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:14.252326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:14.252236 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:14.439833 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:14.439790 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:14.855482 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:14.855447 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:15.044905 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.044855 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:15.234805 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.232254 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:15.406405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.406373 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv"] Nov 02 09:55:15.422431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.422403 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:15.428851 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.427545 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv"] Nov 02 09:55:15.434737 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.434620 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ae13d4-dockercfg-8jj4q\"" Nov 02 09:55:15.434737 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.434707 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ae13d4\"" Nov 02 09:55:15.448987 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.448713 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:15.449976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.449950 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb"] Nov 02 09:55:15.461653 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.461624 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:15.462660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.462630 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb"] Nov 02 09:55:15.557973 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.557741 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv\" (UID: \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:15.557973 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.557791 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sc2b\" (UniqueName: \"kubernetes.io/projected/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-kube-api-access-2sc2b\") pod \"payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv\" (UID: \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:15.557973 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.557911 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jf2\" (UniqueName: \"kubernetes.io/projected/d747555b-d2f7-4d7d-a2be-2828cfe813c5-kube-api-access-v2jf2\") pod \"payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb\" (UID: \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:15.558272 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.558080 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d747555b-d2f7-4d7d-a2be-2828cfe813c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb\" (UID: \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:15.638922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.637040 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:15.660045 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.659996 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv\" (UID: \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:15.660277 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.660076 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sc2b\" (UniqueName: \"kubernetes.io/projected/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-kube-api-access-2sc2b\") pod \"payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv\" (UID: \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:15.660277 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.660191 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jf2\" (UniqueName: \"kubernetes.io/projected/d747555b-d2f7-4d7d-a2be-2828cfe813c5-kube-api-access-v2jf2\") pod \"payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb\" (UID: \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:15.660423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.660374 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d747555b-d2f7-4d7d-a2be-2828cfe813c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb\" (UID: \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:15.661563 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.661539 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d747555b-d2f7-4d7d-a2be-2828cfe813c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb\" (UID: \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:15.662702 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.662679 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv\" (UID: \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:15.707989 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.706721 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jf2\" (UniqueName: \"kubernetes.io/projected/d747555b-d2f7-4d7d-a2be-2828cfe813c5-kube-api-access-v2jf2\") pod \"payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb\" (UID: \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:15.708992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.708965 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sc2b\" (UniqueName: \"kubernetes.io/projected/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-kube-api-access-2sc2b\") pod \"payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv\" (UID: \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:15.760283 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.760247 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:15.781799 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:15.781388 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:16.087639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.087596 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv"] Nov 02 09:55:16.096980 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:55:16.096876 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25989a4c_f1b8_47af_ae2b_bbbf87db2b8e.slice/crio-fcac93c96a2dd2f51c81d2fb1f2930d90d669c8d359bbc5ac3237c2992fdada1 WatchSource:0}: Error finding container fcac93c96a2dd2f51c81d2fb1f2930d90d669c8d359bbc5ac3237c2992fdada1: Status 404 returned error can't find the container with id fcac93c96a2dd2f51c81d2fb1f2930d90d669c8d359bbc5ac3237c2992fdada1 Nov 02 09:55:16.108108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.108029 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb"] Nov 02 09:55:16.118450 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:55:16.118406 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd747555b_d2f7_4d7d_a2be_2828cfe813c5.slice/crio-6308cd66c320855cb36fa16a771ab733e4188079f518228d3be3712aadab4b13 WatchSource:0}: Error finding container 6308cd66c320855cb36fa16a771ab733e4188079f518228d3be3712aadab4b13: Status 404 returned error can't find the container with id 6308cd66c320855cb36fa16a771ab733e4188079f518228d3be3712aadab4b13 Nov 02 09:55:16.197027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.196999 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:16.270503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.270446 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" event={"ID":"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e","Type":"ContainerStarted","Data":"db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c"} Nov 02 09:55:16.270803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.270522 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" event={"ID":"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e","Type":"ContainerStarted","Data":"fcac93c96a2dd2f51c81d2fb1f2930d90d669c8d359bbc5ac3237c2992fdada1"} Nov 02 09:55:16.270803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.270561 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:16.272970 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.272937 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" event={"ID":"d747555b-d2f7-4d7d-a2be-2828cfe813c5","Type":"ContainerStarted","Data":"7f073871f94c20085d822720b25a9f069593d4f0e5d848b85da4cce2ab9fa299"} Nov 02 09:55:16.273108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.272992 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" event={"ID":"d747555b-d2f7-4d7d-a2be-2828cfe813c5","Type":"ContainerStarted","Data":"6308cd66c320855cb36fa16a771ab733e4188079f518228d3be3712aadab4b13"} Nov 02 09:55:16.273108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.273036 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:16.292161 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.292105 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" podStartSLOduration=1.2920881610000001 podStartE2EDuration="1.292088161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:16.288232303 +0000 UTC m=+1195.128174312" watchObservedRunningTime="2026-11-02 09:55:16.292088161 +0000 UTC m=+1195.132030157" Nov 02 09:55:16.310071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.310020 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" podStartSLOduration=1.310003208 podStartE2EDuration="1.310003208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:16.305546326 +0000 UTC m=+1195.145488322" watchObservedRunningTime="2026-11-02 09:55:16.310003208 +0000 UTC m=+1195.149945204" Nov 02 09:55:16.378935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:16.378814 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:17.273552 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:17.273509 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:17.737609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:17.737573 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g"] Nov 02 09:55:17.737922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:17.737854 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" podUID="01a4b738-51a6-448f-a225-3e25ce2c3378" containerName="maas-api" containerID="cri-o://8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd" gracePeriod=30 Nov 02 09:55:17.870673 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:17.870643 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:18.198980 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.196062 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:55:18.213845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.213738 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-metrics-certs\") pod \"01a4b738-51a6-448f-a225-3e25ce2c3378\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " Nov 02 09:55:18.216594 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.216562 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-metrics-certs" pod "01a4b738-51a6-448f-a225-3e25ce2c3378" (UID: "01a4b738-51a6-448f-a225-3e25ce2c3378"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:55:18.223041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.218082 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a4b738-51a6-448f-a225-3e25ce2c3378-kube-api-access-mncsg\" (UniqueName: \"kubernetes.io/projected/01a4b738-51a6-448f-a225-3e25ce2c3378-kube-api-access-mncsg\") pod \"01a4b738-51a6-448f-a225-3e25ce2c3378\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " Nov 02 09:55:18.223041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.218156 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a4b738-51a6-448f-a225-3e25ce2c3378-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a4b738-51a6-448f-a225-3e25ce2c3378-openshift-service-ca\") pod \"01a4b738-51a6-448f-a225-3e25ce2c3378\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " Nov 02 09:55:18.223041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.218229 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-maas-api-tls\") pod \"01a4b738-51a6-448f-a225-3e25ce2c3378\" (UID: \"01a4b738-51a6-448f-a225-3e25ce2c3378\") " Nov 02 09:55:18.223041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.218870 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:18.223041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.219317 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a4b738-51a6-448f-a225-3e25ce2c3378-openshift-service-ca" pod "01a4b738-51a6-448f-a225-3e25ce2c3378" (UID: "01a4b738-51a6-448f-a225-3e25ce2c3378"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:55:18.223041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.222227 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a4b738-51a6-448f-a225-3e25ce2c3378-kube-api-access-mncsg" pod "01a4b738-51a6-448f-a225-3e25ce2c3378" (UID: "01a4b738-51a6-448f-a225-3e25ce2c3378"). InnerVolumeSpecName "kube-api-access-mncsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:55:18.223041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.222667 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-maas-api-tls" pod "01a4b738-51a6-448f-a225-3e25ce2c3378" (UID: "01a4b738-51a6-448f-a225-3e25ce2c3378"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:55:18.287095 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.287057 4467 generic.go:356] "Generic (PLEG): container finished" podID="01a4b738-51a6-448f-a225-3e25ce2c3378" containerID="8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd" exitCode=0 Nov 02 09:55:18.287095 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.287097 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" Nov 02 09:55:18.287600 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.287137 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" event={"ID":"01a4b738-51a6-448f-a225-3e25ce2c3378","Type":"ContainerDied","Data":"8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd"} Nov 02 09:55:18.287600 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.287195 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g" event={"ID":"01a4b738-51a6-448f-a225-3e25ce2c3378","Type":"ContainerDied","Data":"4c75eca72f519caa1e9c5a93e7c25c22a33956c499ccbaccb9fb6b747af7d44d"} Nov 02 09:55:18.287600 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.287216 4467 scope.go:122] "RemoveContainer" containerID="8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd" Nov 02 09:55:18.298360 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.298339 4467 scope.go:122] "RemoveContainer" containerID="8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd" Nov 02 09:55:18.298667 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:18.298645 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd\": container with ID starting with 8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd not found: ID does not exist" containerID="8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd" Nov 02 09:55:18.298788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.298673 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd"} err="failed to get container status \"8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd\": rpc error: code = NotFound desc = could not find container \"8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd\": container with ID starting with 8266f26418bbb45f0dd0c5b04728766fb19e14670a082e148bf49bb1703ed9cd not found: ID does not exist" Nov 02 09:55:18.312137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.312104 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g"] Nov 02 09:55:18.320220 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.320198 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mncsg\" (UniqueName: \"kubernetes.io/projected/01a4b738-51a6-448f-a225-3e25ce2c3378-kube-api-access-mncsg\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:18.320220 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.320219 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a4b738-51a6-448f-a225-3e25ce2c3378-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:18.320412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.320229 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a4b738-51a6-448f-a225-3e25ce2c3378-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:18.320525 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.320503 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ced1b7-77f5445c6d-lpr2g"] Nov 02 09:55:18.492329 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:18.491860 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:19.124703 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:19.124668 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:19.359512 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:19.357533 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a4b738-51a6-448f-a225-3e25ce2c3378" path="/var/lib/kubelet/pods/01a4b738-51a6-448f-a225-3e25ce2c3378/volumes" Nov 02 09:55:19.494404 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:19.494298 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:55:19.786613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:19.786509 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:20.650377 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:20.650339 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:20.829311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:20.829277 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:21.035237 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:21.034306 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:21.243052 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:21.243015 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:21.593508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:21.593454 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:22.214676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:22.214641 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:22.399454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:22.399417 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:22.663466 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:22.663371 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:22.867578 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:22.867536 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:23.055531 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.055446 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:23.236531 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.236496 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c"] Nov 02 09:55:23.237530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.237505 4467 cpu_manager.go:414] "containerMap: removing container" podUID="01a4b738-51a6-448f-a225-3e25ce2c3378" containerName="maas-api" Nov 02 09:55:23.237530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.237529 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a4b738-51a6-448f-a225-3e25ce2c3378" containerName="maas-api" Nov 02 09:55:23.237714 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.237649 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a4b738-51a6-448f-a225-3e25ce2c3378" containerName="maas-api" Nov 02 09:55:23.244027 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.244001 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c"] Nov 02 09:55:23.244174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.244098 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.250589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.250558 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ae13d4-serving-cert\"" Nov 02 09:55:23.347032 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.346992 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:23.373596 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.373568 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-maas-api-tls\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.373740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.373631 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-metrics-certs\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.373920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.373884 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a96cc6a-e2b9-47fc-97ec-4829420592d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.373995 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.373976 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmcq\" (UniqueName: \"kubernetes.io/projected/6a96cc6a-e2b9-47fc-97ec-4829420592d1-kube-api-access-2vmcq\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.475690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.475636 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a96cc6a-e2b9-47fc-97ec-4829420592d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.475886 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.475739 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmcq\" (UniqueName: \"kubernetes.io/projected/6a96cc6a-e2b9-47fc-97ec-4829420592d1-kube-api-access-2vmcq\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.475886 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.475781 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-maas-api-tls\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.475886 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.475863 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-metrics-certs\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.476607 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.476574 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a96cc6a-e2b9-47fc-97ec-4829420592d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.479361 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.479330 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-metrics-certs\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.479462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.479362 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-maas-api-tls\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.500587 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.500552 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmcq\" (UniqueName: \"kubernetes.io/projected/6a96cc6a-e2b9-47fc-97ec-4829420592d1-kube-api-access-2vmcq\") pod \"maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.523386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.523348 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:23.560031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.559995 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:23.716848 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.716816 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:23.726210 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.726173 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk"] Nov 02 09:55:23.727109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.726626 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" containerID="cri-o://d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81" gracePeriod=30 Nov 02 09:55:23.851285 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:55:23.851257 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a96cc6a_e2b9_47fc_97ec_4829420592d1.slice/crio-876b761272c84a12c172b4ee57b2811e67aa0d318d9c026ba588392abf7ebf69 WatchSource:0}: Error finding container 876b761272c84a12c172b4ee57b2811e67aa0d318d9c026ba588392abf7ebf69: Status 404 returned error can't find the container with id 876b761272c84a12c172b4ee57b2811e67aa0d318d9c026ba588392abf7ebf69 Nov 02 09:55:23.852040 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.852012 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c"] Nov 02 09:55:23.898671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:23.898598 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:24.363334 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:24.363297 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" event={"ID":"6a96cc6a-e2b9-47fc-97ec-4829420592d1","Type":"ContainerStarted","Data":"876b761272c84a12c172b4ee57b2811e67aa0d318d9c026ba588392abf7ebf69"} Nov 02 09:55:24.649801 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:24.649221 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:25.399716 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:25.399664 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" event={"ID":"6a96cc6a-e2b9-47fc-97ec-4829420592d1","Type":"ContainerStarted","Data":"f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8"} Nov 02 09:55:25.400307 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:25.399756 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:25.422017 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:25.421973 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:25.425165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:25.425106 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" podStartSLOduration=1.834962484 podStartE2EDuration="2.425089291s" totalImagesPullingTime="590.126807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:25.421331552 +0000 UTC m=+1204.261273571" watchObservedRunningTime="2026-11-02 09:55:25.425089291 +0000 UTC m=+1204.265031287" Nov 02 09:55:26.348386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:26.348350 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:26.710172 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:26.710085 4467 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 09:55:26.710172 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:26.710147 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 09:55:26.855451 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:26.855411 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:55:27.134358 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:27.134268 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:27.283369 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:27.283339 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:55:27.284249 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:27.284225 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:55:27.392730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:27.392629 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:27.590384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:27.590351 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:28.407551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:28.407518 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:28.649820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:28.649430 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:29.017347 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.017277 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:29.144971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.144939 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-envoy\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145178 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.144998 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-token\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-token\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145178 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145046 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-podinfo\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145178 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145108 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-kube-api-access-5hfpd\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-kube-api-access-5hfpd\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145178 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145142 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istiod-ca-cert\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145178 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145169 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-credential-socket\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145432 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145236 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-certs\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145487 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145467 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-data\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145540 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145504 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-socket\") pod \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\" (UID: \"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1\") " Nov 02 09:55:29.145656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145621 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-credential-socket" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:55:29.146013 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.145988 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istiod-ca-cert" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:55:29.146139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.146121 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.146217 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.146147 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.146281 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.146219 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-certs" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:55:29.146336 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.146320 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-data" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:55:29.146391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.146354 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-socket" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:55:29.147378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.147358 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-envoy" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:55:29.147503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.147486 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-podinfo" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:55:29.147713 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.147693 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-token" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:55:29.147713 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.147705 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-kube-api-access-5hfpd" pod "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" (UID: "c39ac5fb-6f24-453e-99ec-a0bdf072b4d1"). InnerVolumeSpecName "kube-api-access-5hfpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:55:29.247346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.247315 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.247346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.247343 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.247346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.247353 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.247581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.247362 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hfpd\" (UniqueName: \"kubernetes.io/projected/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-kube-api-access-5hfpd\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.247581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.247371 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.247581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.247380 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.247581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.247388 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:29.361230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.361197 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:29.434028 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.433987 4467 generic.go:356] "Generic (PLEG): container finished" podID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerID="d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81" exitCode=0 Nov 02 09:55:29.434457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.434040 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" Nov 02 09:55:29.434457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.434043 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" event={"ID":"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1","Type":"ContainerDied","Data":"d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81"} Nov 02 09:55:29.434457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.434096 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk" event={"ID":"c39ac5fb-6f24-453e-99ec-a0bdf072b4d1","Type":"ContainerDied","Data":"910743d761d3c237adef81494b2ac064333c7cca5e93af514a65d204ecddf986"} Nov 02 09:55:29.434457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.434129 4467 scope.go:122] "RemoveContainer" containerID="d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81" Nov 02 09:55:29.444410 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.444388 4467 scope.go:122] "RemoveContainer" containerID="d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81" Nov 02 09:55:29.444701 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:29.444680 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81\": container with ID starting with d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81 not found: ID does not exist" containerID="d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81" Nov 02 09:55:29.444772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.444710 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81"} err="failed to get container status \"d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81\": rpc error: code = NotFound desc = could not find container \"d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81\": container with ID starting with d7f2ea7c5c69c94c1fdde8a9bc8fb2fd7a80633a28788a11fe892d5b595b7f81 not found: ID does not exist" Nov 02 09:55:29.454412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.454387 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk"] Nov 02 09:55:29.458390 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:29.458368 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-g2dvk"] Nov 02 09:55:30.262921 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:30.262843 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:31.348959 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:31.347106 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:31.354352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:31.354317 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" path="/var/lib/kubelet/pods/c39ac5fb-6f24-453e-99ec-a0bdf072b4d1/volumes" Nov 02 09:55:31.414234 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:31.414202 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:55:32.166043 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:32.166004 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:33.503206 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:33.503003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:33.608915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:33.608051 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:55:33.608915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:33.608131 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:55:34.215929 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:34.211869 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:34.898840 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:34.897437 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:35.460109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:35.460077 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:35.641121 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:35.641077 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:35.850215 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:35.850189 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:36.030847 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:36.030809 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:36.683878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:36.683824 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:36.871444 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:36.871412 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:37.053721 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:37.053613 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:37.391652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:37.391618 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:55:37.833517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:37.833478 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:37.978914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:37.929687 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd"] Nov 02 09:55:38.172776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.172679 4467 cpu_manager.go:414] "containerMap: removing container" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" Nov 02 09:55:38.172776 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.172712 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" Nov 02 09:55:38.173429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.172929 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39ac5fb-6f24-453e-99ec-a0bdf072b4d1" containerName="istio-proxy" Nov 02 09:55:38.178566 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.178535 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd"] Nov 02 09:55:38.178948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.178922 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.287017 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.286984 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.287259 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.287238 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.287384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.287365 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.287539 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.287523 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.287656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.287639 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.287785 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.287768 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfn9h\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-kube-api-access-bfn9h\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.287924 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.287882 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.288160 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.288142 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d67c593-7285-4756-a5b3-43529b70ae5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.288281 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.288266 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394502 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394559 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394588 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394643 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394668 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394708 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfn9h\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-kube-api-access-bfn9h\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394736 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394850 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d67c593-7285-4756-a5b3-43529b70ae5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.394881 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.395309 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.395352 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.395686 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.396006 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.399695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.397974 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d67c593-7285-4756-a5b3-43529b70ae5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.410562 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.410533 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.421260 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.421233 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.422289 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.422037 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfn9h\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-kube-api-access-bfn9h\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.428212 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.428144 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7prvd\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.591448 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.591413 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:38.820308 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:38.820268 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:39.825130 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:39.825098 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd"] Nov 02 09:55:39.833670 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:39.833629 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:39.837952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:39.837910 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:55:39.838084 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:39.838010 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:55:39.838862 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:39.838832 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 09:55:39.843197 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:39.843169 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" event={"ID":"5d67c593-7285-4756-a5b3-43529b70ae5a","Type":"ContainerStarted","Data":"f0b328ae11934e48044d264d70e1ef73359baf6e9de00e6ff42075b3909791d0"} Nov 02 09:55:40.824727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:40.824692 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:40.850581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:40.850550 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" event={"ID":"5d67c593-7285-4756-a5b3-43529b70ae5a","Type":"ContainerStarted","Data":"ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d"} Nov 02 09:55:40.875763 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:40.875719 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podStartSLOduration=3.875704276 podStartE2EDuration="3.875704276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:40.87561845 +0000 UTC m=+1219.715560445" watchObservedRunningTime="2026-11-02 09:55:40.875704276 +0000 UTC m=+1219.715646268" Nov 02 09:55:41.595831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:41.595751 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:41.596257 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:41.596234 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 09:55:41.596415 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:41.596392 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 09:55:41.876967 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:41.875638 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:42.596916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:42.592388 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 09:55:42.596916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:42.592444 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 09:55:42.848365 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:42.847011 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:43.607330 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:43.607290 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 09:55:43.607748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:43.607359 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 09:55:43.931012 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:43.930749 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:44.612947 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:44.606025 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 09:55:44.612947 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:44.606083 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 09:55:45.054671 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:45.054585 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:45.402108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:45.402070 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:45.601916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:45.597267 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 09:55:45.601916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:45.597329 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 09:55:45.628386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:45.628320 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:45.823397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:45.823354 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:46.458271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:46.458239 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:46.592590 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:46.592561 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 09:55:46.592754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:46.592615 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 09:55:47.239392 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.239332 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:47.426795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.426760 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:47.592758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.592721 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 09:55:47.592953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.592782 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 09:55:47.742219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.742160 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz"] Nov 02 09:55:47.743378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.743319 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" podUID="13d82ead-327a-420f-82a2-efe567b10bda" containerName="maas-api" containerID="cri-o://4217a91eabe1c913bb1ce5156e600c3186d7112ce7a7fc27db6d45a0415d84eb" gracePeriod=30 Nov 02 09:55:47.874376 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.874331 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:47.963397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.963273 4467 generic.go:356] "Generic (PLEG): container finished" podID="13d82ead-327a-420f-82a2-efe567b10bda" containerID="4217a91eabe1c913bb1ce5156e600c3186d7112ce7a7fc27db6d45a0415d84eb" exitCode=0 Nov 02 09:55:47.963397 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:47.963316 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" event={"ID":"13d82ead-327a-420f-82a2-efe567b10bda","Type":"ContainerDied","Data":"4217a91eabe1c913bb1ce5156e600c3186d7112ce7a7fc27db6d45a0415d84eb"} Nov 02 09:55:48.126404 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.126326 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:55:48.199912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.195400 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-metrics-certs\") pod \"13d82ead-327a-420f-82a2-efe567b10bda\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " Nov 02 09:55:48.199912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.195496 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d82ead-327a-420f-82a2-efe567b10bda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d82ead-327a-420f-82a2-efe567b10bda-openshift-service-ca\") pod \"13d82ead-327a-420f-82a2-efe567b10bda\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " Nov 02 09:55:48.199912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.195517 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-maas-api-tls\") pod \"13d82ead-327a-420f-82a2-efe567b10bda\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " Nov 02 09:55:48.199912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.195991 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d82ead-327a-420f-82a2-efe567b10bda-kube-api-access-4p8w7\" (UniqueName: \"kubernetes.io/projected/13d82ead-327a-420f-82a2-efe567b10bda-kube-api-access-4p8w7\") pod \"13d82ead-327a-420f-82a2-efe567b10bda\" (UID: \"13d82ead-327a-420f-82a2-efe567b10bda\") " Nov 02 09:55:48.199912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.197450 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d82ead-327a-420f-82a2-efe567b10bda-openshift-service-ca" pod "13d82ead-327a-420f-82a2-efe567b10bda" (UID: "13d82ead-327a-420f-82a2-efe567b10bda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:55:48.203517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.203488 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-metrics-certs" pod "13d82ead-327a-420f-82a2-efe567b10bda" (UID: "13d82ead-327a-420f-82a2-efe567b10bda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:55:48.207353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.207327 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-maas-api-tls" pod "13d82ead-327a-420f-82a2-efe567b10bda" (UID: "13d82ead-327a-420f-82a2-efe567b10bda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:55:48.210424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.210397 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d82ead-327a-420f-82a2-efe567b10bda-kube-api-access-4p8w7" pod "13d82ead-327a-420f-82a2-efe567b10bda" (UID: "13d82ead-327a-420f-82a2-efe567b10bda"). InnerVolumeSpecName "kube-api-access-4p8w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:55:48.299984 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.298219 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d82ead-327a-420f-82a2-efe567b10bda-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:48.299984 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.299937 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:48.299984 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.299973 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p8w7\" (UniqueName: \"kubernetes.io/projected/13d82ead-327a-420f-82a2-efe567b10bda-kube-api-access-4p8w7\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:48.299984 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.299991 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d82ead-327a-420f-82a2-efe567b10bda-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:55:48.771778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.771498 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:48.971248 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.971219 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" Nov 02 09:55:48.971453 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.971255 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz" event={"ID":"13d82ead-327a-420f-82a2-efe567b10bda","Type":"ContainerDied","Data":"c75cef81ede910db92f59280091d67ba2498c089c88b29aa56f4da81815b83a1"} Nov 02 09:55:48.971453 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:48.971316 4467 scope.go:122] "RemoveContainer" containerID="4217a91eabe1c913bb1ce5156e600c3186d7112ce7a7fc27db6d45a0415d84eb" Nov 02 09:55:49.001032 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.000969 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz"] Nov 02 09:55:49.006245 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.006218 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a71884-758b75998b-z2fmz"] Nov 02 09:55:49.020528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.020499 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:49.338761 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.338723 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d82ead-327a-420f-82a2-efe567b10bda" path="/var/lib/kubelet/pods/13d82ead-327a-420f-82a2-efe567b10bda/volumes" Nov 02 09:55:49.593065 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.592965 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:55:49.593065 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.593036 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:55:49.633036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.633001 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:49.633471 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.633450 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:49.634774 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.634754 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 09:55:49.774040 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:49.774002 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:50.752419 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:50.752385 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:51.789254 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:51.789218 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:51.967778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:51.967743 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:52.220179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:52.220043 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:52.410811 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:52.410750 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:52.600293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:52.600249 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:52.843999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:52.843947 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:53.026649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:53.026571 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:53.222942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:53.221446 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:53.418912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:53.418548 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:53.674994 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:53.670865 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:53.902994 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:53.902481 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:54.241091 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:54.241052 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:54.431475 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:54.431327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:54.612743 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:54.612706 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:54.815385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:54.815349 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:55.014988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.013632 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:55.174848 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.174796 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9"] Nov 02 09:55:55.176056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.176032 4467 cpu_manager.go:414] "containerMap: removing container" podUID="13d82ead-327a-420f-82a2-efe567b10bda" containerName="maas-api" Nov 02 09:55:55.176279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.176263 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d82ead-327a-420f-82a2-efe567b10bda" containerName="maas-api" Nov 02 09:55:55.176589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.176572 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d82ead-327a-420f-82a2-efe567b10bda" containerName="maas-api" Nov 02 09:55:55.186400 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.186378 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9"] Nov 02 09:55:55.212236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.211277 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.241939 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.239274 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\"" Nov 02 09:55:55.241939 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.239396 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-941f3356d954\"" Nov 02 09:55:55.241939 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.239494 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-msn5j\"" Nov 02 09:55:55.247346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.247211 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\"" Nov 02 09:55:55.247346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.247297 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\"" Nov 02 09:55:55.334360 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.334298 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\"" Nov 02 09:55:55.376386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.376331 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.376670 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.376649 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.376823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.376804 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.376991 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.376973 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjldc\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-kube-api-access-kjldc\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.489914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.489077 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.489914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.489146 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.489914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.489191 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.489914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.489238 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjldc\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-kube-api-access-kjldc\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.493913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.490474 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.501913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.499404 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.520919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.520467 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.553917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.552875 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjldc\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-kube-api-access-kjldc\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:55.761663 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.761550 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:55.806283 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:55.806244 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:56.234919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:56.213103 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:56.798016 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:56.797978 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9"] Nov 02 09:55:57.077266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.077228 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:57.122990 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:57.122884 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:55:57.523479 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.523331 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:57.614585 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.605482 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" event={"ID":"cdd86db9-3ec8-4db4-8a32-4053926cb17b","Type":"ContainerStarted","Data":"829e08a626e316c8ff93ed68077c84dc4fc887e21b282c4dffc2c3a5bf43bd06"} Nov 02 09:55:57.855915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.846510 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7"] Nov 02 09:55:57.880101 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.880076 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:57.940915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.922293 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"e2e-cfl-89ee55-provider-cred\"" Nov 02 09:55:57.961731 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.961473 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7"] Nov 02 09:55:57.976946 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.975951 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:57.977357 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.977319 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvxt\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-kube-api-access-dpvxt\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:57.977535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.977517 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:57.977678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:57.977663 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.111915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.111791 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.112119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.111974 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvxt\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-kube-api-access-dpvxt\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.112119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.112017 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.112119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.112059 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.117911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.113673 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.117911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.114401 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.152019 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.151867 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:55:58.171924 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.158617 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.175681 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.175618 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvxt\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-kube-api-access-dpvxt\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.441808 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.423417 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:55:58.625037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.623080 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" event={"ID":"cdd86db9-3ec8-4db4-8a32-4053926cb17b","Type":"ContainerStarted","Data":"c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771"} Nov 02 09:55:58.625037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.624276 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:55:58.642842 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:58.642485 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-941f3356d954: secret "agc-cred-941f3356d954" not found Nov 02 09:55:58.642842 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:58.642512 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9: secret "agc-cred-941f3356d954" not found Nov 02 09:55:58.642842 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:58.642585 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials podName:cdd86db9-3ec8-4db4-8a32-4053926cb17b nodeName:}" failed. No retries permitted until 2026-11-02 09:55:59.142563688 +0000 UTC m=+1237.982505681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b") : secret "agc-cred-941f3356d954" not found Nov 02 09:55:58.796009 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.795909 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" podStartSLOduration=3.795878943 podStartE2EDuration="3.795878943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:58.665254369 +0000 UTC m=+1237.505196365" watchObservedRunningTime="2026-11-02 09:55:58.795878943 +0000 UTC m=+1237.635820940" Nov 02 09:55:58.797256 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:58.797230 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7"] Nov 02 09:55:59.160581 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:59.160548 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-941f3356d954: secret "agc-cred-941f3356d954" not found Nov 02 09:55:59.160581 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:59.160576 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9: secret "agc-cred-941f3356d954" not found Nov 02 09:55:59.160792 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:55:59.160628 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials podName:cdd86db9-3ec8-4db4-8a32-4053926cb17b nodeName:}" failed. No retries permitted until 2026-11-02 09:56:00.1606137 +0000 UTC m=+1239.000555673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b") : secret "agc-cred-941f3356d954" not found Nov 02 09:55:59.631932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:59.631862 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" event={"ID":"bdc2ab2e-9620-4a5f-be39-10b70a1775cd","Type":"ContainerStarted","Data":"f156dd88934171584404279c2cc40618bb605dd195f71b3bcd7f434d66aeca22"} Nov 02 09:55:59.632401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:59.631958 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" event={"ID":"bdc2ab2e-9620-4a5f-be39-10b70a1775cd","Type":"ContainerStarted","Data":"3655990991b8b4d54e92620969bc0701c1a448fe119621eeec9c8f3e5996db0e"} Nov 02 09:55:59.654250 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:59.654199 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" podStartSLOduration=2.654185508 podStartE2EDuration="2.654185508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:59.651697328 +0000 UTC m=+1238.491639324" watchObservedRunningTime="2026-11-02 09:55:59.654185508 +0000 UTC m=+1238.494127557" Nov 02 09:55:59.957148 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:55:59.957043 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:00.174372 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:00.174343 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-941f3356d954: secret "agc-cred-941f3356d954" not found Nov 02 09:56:00.174372 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:00.174368 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9: secret "agc-cred-941f3356d954" not found Nov 02 09:56:00.174580 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:00.174422 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials podName:cdd86db9-3ec8-4db4-8a32-4053926cb17b nodeName:}" failed. No retries permitted until 2026-11-02 09:56:02.174408212 +0000 UTC m=+1241.014350185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b") : secret "agc-cred-941f3356d954" not found Nov 02 09:56:00.565756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:00.565721 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb"] Nov 02 09:56:00.592735 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:00.592704 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv"] Nov 02 09:56:00.652300 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:00.652271 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:56:00.733083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:00.733051 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:00.916415 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:00.916317 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:01.116140 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:01.116107 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:01.338650 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:01.338616 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:02.155214 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:02.153779 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:02.214193 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:02.214160 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-941f3356d954: secret "agc-cred-941f3356d954" not found Nov 02 09:56:02.214193 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:02.214186 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9: secret "agc-cred-941f3356d954" not found Nov 02 09:56:02.214418 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:02.214239 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials podName:cdd86db9-3ec8-4db4-8a32-4053926cb17b nodeName:}" failed. No retries permitted until 2026-11-02 09:56:06.214225128 +0000 UTC m=+1245.054167104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b") : secret "agc-cred-941f3356d954" not found Nov 02 09:56:02.932910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:02.930423 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:03.271010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:03.270850 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:03.748162 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:03.748132 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:03.932229 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:03.932181 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:04.559653 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:04.559611 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:05.248447 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:05.248417 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:05.567063 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:05.567008 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" podUID="d747555b-d2f7-4d7d-a2be-2828cfe813c5" containerName="payload-processing" containerID="cri-o://7f073871f94c20085d822720b25a9f069593d4f0e5d848b85da4cce2ab9fa299" gracePeriod=25 Nov 02 09:56:05.579256 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:05.579227 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:56:05.593986 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:05.593948 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" podUID="25989a4c-f1b8-47af-ae2b-bbbf87db2b8e" containerName="payload-pre-processing" containerID="cri-o://db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c" gracePeriod=25 Nov 02 09:56:05.686202 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:05.686174 4467 generic.go:356] "Generic (PLEG): container finished" podID="d747555b-d2f7-4d7d-a2be-2828cfe813c5" containerID="7f073871f94c20085d822720b25a9f069593d4f0e5d848b85da4cce2ab9fa299" exitCode=0 Nov 02 09:56:05.686322 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:05.686259 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" event={"ID":"d747555b-d2f7-4d7d-a2be-2828cfe813c5","Type":"ContainerDied","Data":"7f073871f94c20085d822720b25a9f069593d4f0e5d848b85da4cce2ab9fa299"} Nov 02 09:56:05.893899 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:05.893857 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:56:05.963530 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:05.963493 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:06.004850 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.004819 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:56:06.090137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.090101 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d747555b-d2f7-4d7d-a2be-2828cfe813c5-kube-api-access-v2jf2\" (UniqueName: \"kubernetes.io/projected/d747555b-d2f7-4d7d-a2be-2828cfe813c5-kube-api-access-v2jf2\") pod \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\" (UID: \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\") " Nov 02 09:56:06.090301 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.090150 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d747555b-d2f7-4d7d-a2be-2828cfe813c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d747555b-d2f7-4d7d-a2be-2828cfe813c5-plugins-config-volume\") pod \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\" (UID: \"d747555b-d2f7-4d7d-a2be-2828cfe813c5\") " Nov 02 09:56:06.090552 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.090525 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d747555b-d2f7-4d7d-a2be-2828cfe813c5-plugins-config-volume" pod "d747555b-d2f7-4d7d-a2be-2828cfe813c5" (UID: "d747555b-d2f7-4d7d-a2be-2828cfe813c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:56:06.092352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.092326 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d747555b-d2f7-4d7d-a2be-2828cfe813c5-kube-api-access-v2jf2" pod "d747555b-d2f7-4d7d-a2be-2828cfe813c5" (UID: "d747555b-d2f7-4d7d-a2be-2828cfe813c5"). InnerVolumeSpecName "kube-api-access-v2jf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:56:06.195756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.194350 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-plugins-config-volume\") pod \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\" (UID: \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\") " Nov 02 09:56:06.195756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.194464 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-kube-api-access-2sc2b\" (UniqueName: \"kubernetes.io/projected/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-kube-api-access-2sc2b\") pod \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\" (UID: \"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e\") " Nov 02 09:56:06.195756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.195286 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jf2\" (UniqueName: \"kubernetes.io/projected/d747555b-d2f7-4d7d-a2be-2828cfe813c5-kube-api-access-v2jf2\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:06.195756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.195310 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d747555b-d2f7-4d7d-a2be-2828cfe813c5-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:06.198084 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.198036 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-kube-api-access-2sc2b" pod "25989a4c-f1b8-47af-ae2b-bbbf87db2b8e" (UID: "25989a4c-f1b8-47af-ae2b-bbbf87db2b8e"). InnerVolumeSpecName "kube-api-access-2sc2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:56:06.205150 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.205095 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-plugins-config-volume" pod "25989a4c-f1b8-47af-ae2b-bbbf87db2b8e" (UID: "25989a4c-f1b8-47af-ae2b-bbbf87db2b8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:56:06.297775 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.297726 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:06.297775 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.297767 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sc2b\" (UniqueName: \"kubernetes.io/projected/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e-kube-api-access-2sc2b\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:06.298073 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:06.297907 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-941f3356d954: secret "agc-cred-941f3356d954" not found Nov 02 09:56:06.298073 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:06.297934 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9: secret "agc-cred-941f3356d954" not found Nov 02 09:56:06.298073 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:06.297996 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials podName:cdd86db9-3ec8-4db4-8a32-4053926cb17b nodeName:}" failed. No retries permitted until 2026-11-02 09:56:14.297977299 +0000 UTC m=+1253.137919293 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b") : secret "agc-cred-941f3356d954" not found Nov 02 09:56:06.795058 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.770056 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c"] Nov 02 09:56:06.795610 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.795364 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" podUID="6a96cc6a-e2b9-47fc-97ec-4829420592d1" containerName="maas-api" containerID="cri-o://f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8" gracePeriod=30 Nov 02 09:56:07.044912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:06.995030 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:07.143917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.127084 4467 generic.go:356] "Generic (PLEG): container finished" podID="25989a4c-f1b8-47af-ae2b-bbbf87db2b8e" containerID="db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c" exitCode=0 Nov 02 09:56:07.143917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.127396 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" Nov 02 09:56:07.143917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.131528 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" event={"ID":"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e","Type":"ContainerDied","Data":"db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c"} Nov 02 09:56:07.143917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.131618 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv" event={"ID":"25989a4c-f1b8-47af-ae2b-bbbf87db2b8e","Type":"ContainerDied","Data":"fcac93c96a2dd2f51c81d2fb1f2930d90d669c8d359bbc5ac3237c2992fdada1"} Nov 02 09:56:07.143917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.131652 4467 scope.go:122] "RemoveContainer" containerID="db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c" Nov 02 09:56:07.152706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.152670 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" event={"ID":"d747555b-d2f7-4d7d-a2be-2828cfe813c5","Type":"ContainerDied","Data":"6308cd66c320855cb36fa16a771ab733e4188079f518228d3be3712aadab4b13"} Nov 02 09:56:07.153088 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.153049 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb" Nov 02 09:56:07.483935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.479739 4467 scope.go:122] "RemoveContainer" containerID="db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c" Nov 02 09:56:07.521800 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:07.521600 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c\": container with ID starting with db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c not found: ID does not exist" containerID="db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c" Nov 02 09:56:07.521800 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.521756 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c"} err="failed to get container status \"db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c\": rpc error: code = NotFound desc = could not find container \"db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c\": container with ID starting with db7e7a9b65a6d88617090f571907c44f713584bb062c18e69e6a7dd2c6333d9c not found: ID does not exist" Nov 02 09:56:07.522081 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.521884 4467 scope.go:122] "RemoveContainer" containerID="7f073871f94c20085d822720b25a9f069593d4f0e5d848b85da4cce2ab9fa299" Nov 02 09:56:07.772831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.772748 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv"] Nov 02 09:56:07.773098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.773079 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae13d4-8f94f446c-jcvtv"] Nov 02 09:56:07.773230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.773217 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb"] Nov 02 09:56:07.786024 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.785996 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae13d4-6d88f49fd5-bjfjb"] Nov 02 09:56:07.894844 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.894588 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:56:07.958286 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.958242 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a96cc6a-e2b9-47fc-97ec-4829420592d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a96cc6a-e2b9-47fc-97ec-4829420592d1-openshift-service-ca\") pod \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " Nov 02 09:56:07.959419 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.959276 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a96cc6a-e2b9-47fc-97ec-4829420592d1-openshift-service-ca" pod "6a96cc6a-e2b9-47fc-97ec-4829420592d1" (UID: "6a96cc6a-e2b9-47fc-97ec-4829420592d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:56:07.959803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.959779 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a96cc6a-e2b9-47fc-97ec-4829420592d1-kube-api-access-2vmcq\" (UniqueName: \"kubernetes.io/projected/6a96cc6a-e2b9-47fc-97ec-4829420592d1-kube-api-access-2vmcq\") pod \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " Nov 02 09:56:07.960424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.960402 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-maas-api-tls\") pod \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " Nov 02 09:56:07.960954 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.960935 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-metrics-certs\") pod \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\" (UID: \"6a96cc6a-e2b9-47fc-97ec-4829420592d1\") " Nov 02 09:56:07.962168 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.962146 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a96cc6a-e2b9-47fc-97ec-4829420592d1-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:07.962437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.962413 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a96cc6a-e2b9-47fc-97ec-4829420592d1-kube-api-access-2vmcq" pod "6a96cc6a-e2b9-47fc-97ec-4829420592d1" (UID: "6a96cc6a-e2b9-47fc-97ec-4829420592d1"). InnerVolumeSpecName "kube-api-access-2vmcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:56:07.962698 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.962679 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-maas-api-tls" pod "6a96cc6a-e2b9-47fc-97ec-4829420592d1" (UID: "6a96cc6a-e2b9-47fc-97ec-4829420592d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:56:07.963717 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:07.963683 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-metrics-certs" pod "6a96cc6a-e2b9-47fc-97ec-4829420592d1" (UID: "6a96cc6a-e2b9-47fc-97ec-4829420592d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:56:08.008043 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.007850 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:08.077930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.066069 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vmcq\" (UniqueName: \"kubernetes.io/projected/6a96cc6a-e2b9-47fc-97ec-4829420592d1-kube-api-access-2vmcq\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:08.077930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.066108 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:08.077930 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.066125 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a96cc6a-e2b9-47fc-97ec-4829420592d1-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:08.165636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.165515 4467 generic.go:356] "Generic (PLEG): container finished" podID="6a96cc6a-e2b9-47fc-97ec-4829420592d1" containerID="f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8" exitCode=0 Nov 02 09:56:08.165636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.165585 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" Nov 02 09:56:08.165636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.165609 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" event={"ID":"6a96cc6a-e2b9-47fc-97ec-4829420592d1","Type":"ContainerDied","Data":"f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8"} Nov 02 09:56:08.165941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.165673 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c" event={"ID":"6a96cc6a-e2b9-47fc-97ec-4829420592d1","Type":"ContainerDied","Data":"876b761272c84a12c172b4ee57b2811e67aa0d318d9c026ba588392abf7ebf69"} Nov 02 09:56:08.165941 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.165691 4467 scope.go:122] "RemoveContainer" containerID="f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8" Nov 02 09:56:08.181312 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.181292 4467 scope.go:122] "RemoveContainer" containerID="f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8" Nov 02 09:56:08.184500 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:08.181707 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8\": container with ID starting with f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8 not found: ID does not exist" containerID="f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8" Nov 02 09:56:08.184500 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.181742 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8"} err="failed to get container status \"f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8\": rpc error: code = NotFound desc = could not find container \"f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8\": container with ID starting with f49b723586ad4092e3653ee08a2d8bf93cf1420ecc01a881fe0f4fa0ec7402e8 not found: ID does not exist" Nov 02 09:56:08.201881 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.198061 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:08.205125 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.205067 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c"] Nov 02 09:56:08.210380 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.210351 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae13d4-8d4bcfcdb-vv27c"] Nov 02 09:56:08.545427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.545339 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:08.728912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.728266 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:08.936914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:08.927758 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:09.168323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:09.145876 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:09.470949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:09.470245 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25989a4c-f1b8-47af-ae2b-bbbf87db2b8e" path="/var/lib/kubelet/pods/25989a4c-f1b8-47af-ae2b-bbbf87db2b8e/volumes" Nov 02 09:56:09.471169 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:09.471125 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a96cc6a-e2b9-47fc-97ec-4829420592d1" path="/var/lib/kubelet/pods/6a96cc6a-e2b9-47fc-97ec-4829420592d1/volumes" Nov 02 09:56:09.473747 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:09.472953 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d747555b-d2f7-4d7d-a2be-2828cfe813c5" path="/var/lib/kubelet/pods/d747555b-d2f7-4d7d-a2be-2828cfe813c5/volumes" Nov 02 09:56:10.166914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:10.143813 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:10.327167 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:10.327066 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:10.555978 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:10.555378 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:10.656914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:10.656196 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:56:11.410119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:11.410082 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:11.665058 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:11.664978 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:56:11.755066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:11.755026 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9"] Nov 02 09:56:11.925513 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:11.925410 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:13.123687 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:13.123652 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:13.842754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:13.842721 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:14.303599 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:14.303516 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-941f3356d954: secret "agc-cred-941f3356d954" not found Nov 02 09:56:14.303599 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:14.303542 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9: secret "agc-cred-941f3356d954" not found Nov 02 09:56:14.304109 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:14.303618 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials podName:cdd86db9-3ec8-4db4-8a32-4053926cb17b nodeName:}" failed. No retries permitted until 2026-11-02 09:56:30.303598996 +0000 UTC m=+1269.143540982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b") : secret "agc-cred-941f3356d954" not found Nov 02 09:56:14.550181 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:14.549024 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:15.499557 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:15.499526 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:16.155952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:16.155917 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:16.661739 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:16.661703 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:16.755739 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:16.755703 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" podUID="cdd86db9-3ec8-4db4-8a32-4053926cb17b" containerName="payload-processing" containerID="cri-o://c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771" gracePeriod=25 Nov 02 09:56:16.870914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:16.852463 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:17.067971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.067362 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:17.163412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.163390 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:56:17.215912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.215814 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-routing-overlay-volume\") pod \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " Nov 02 09:56:17.216071 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.215996 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials\") pod \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " Nov 02 09:56:17.216130 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.216113 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-kube-api-access-kjldc\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-kube-api-access-kjldc\") pod \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " Nov 02 09:56:17.216186 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.216145 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-plugins-config-volume\") pod \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\" (UID: \"cdd86db9-3ec8-4db4-8a32-4053926cb17b\") " Nov 02 09:56:17.216670 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.216295 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-routing-overlay-volume" pod "cdd86db9-3ec8-4db4-8a32-4053926cb17b" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:56:17.216670 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.216633 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-plugins-config-volume" pod "cdd86db9-3ec8-4db4-8a32-4053926cb17b" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:56:17.216839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.216697 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:17.218536 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.218502 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-kube-api-access-kjldc" pod "cdd86db9-3ec8-4db4-8a32-4053926cb17b" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b"). InnerVolumeSpecName "kube-api-access-kjldc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:56:17.218706 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.218683 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials" pod "cdd86db9-3ec8-4db4-8a32-4053926cb17b" (UID: "cdd86db9-3ec8-4db4-8a32-4053926cb17b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:56:17.289836 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.289810 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:17.318391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.318351 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjldc\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-kube-api-access-kjldc\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:17.318391 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.318390 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd86db9-3ec8-4db4-8a32-4053926cb17b-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:17.318601 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.318407 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdd86db9-3ec8-4db4-8a32-4053926cb17b-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:56:17.410238 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.410199 4467 generic.go:356] "Generic (PLEG): container finished" podID="cdd86db9-3ec8-4db4-8a32-4053926cb17b" containerID="c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771" exitCode=0 Nov 02 09:56:17.410401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.410246 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" Nov 02 09:56:17.410401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.410283 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" event={"ID":"cdd86db9-3ec8-4db4-8a32-4053926cb17b","Type":"ContainerDied","Data":"c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771"} Nov 02 09:56:17.410401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.410340 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9" event={"ID":"cdd86db9-3ec8-4db4-8a32-4053926cb17b","Type":"ContainerDied","Data":"829e08a626e316c8ff93ed68077c84dc4fc887e21b282c4dffc2c3a5bf43bd06"} Nov 02 09:56:17.410401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.410369 4467 scope.go:122] "RemoveContainer" containerID="c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771" Nov 02 09:56:17.420410 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.420389 4467 scope.go:122] "RemoveContainer" containerID="c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771" Nov 02 09:56:17.420716 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:17.420688 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771\": container with ID starting with c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771 not found: ID does not exist" containerID="c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771" Nov 02 09:56:17.420788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.420726 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771"} err="failed to get container status \"c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771\": rpc error: code = NotFound desc = could not find container \"c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771\": container with ID starting with c80edf85518b7bcc69e243e042e06966fd763c151f7bb3df4fb3081b96904771 not found: ID does not exist" Nov 02 09:56:17.429581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.429561 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9"] Nov 02 09:56:17.435215 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.435195 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvqjj9"] Nov 02 09:56:17.748466 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:17.748426 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:18.565230 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:18.565198 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:19.286873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:19.286839 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:19.333683 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:19.333652 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd86db9-3ec8-4db4-8a32-4053926cb17b" path="/var/lib/kubelet/pods/cdd86db9-3ec8-4db4-8a32-4053926cb17b/volumes" Nov 02 09:56:19.954679 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:19.954638 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:20.684651 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:20.684605 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:21.625916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:21.624331 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:22.668845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:22.668798 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:23.374667 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:23.374639 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:23.860555 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:23.860514 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:24.052680 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:24.052646 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:24.230406 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:24.230302 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:24.870927 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:24.870880 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:25.570463 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:25.570420 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:26.262534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:26.262500 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:26.856342 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:26.856309 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:56:26.952607 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:26.952408 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:27.635841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:27.635742 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:28.377606 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:28.377576 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:29.029617 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:29.029590 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:29.747124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:29.747093 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:29.974989 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:29.974936 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:56:30.426035 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:30.426003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:31.062384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:31.062285 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:31.770187 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:31.770153 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:32.439221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:32.439180 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:33.049915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:33.049766 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:33.441723 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:33.441685 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:34.141503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.141464 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:34.337160 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.337108 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:34.518606 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.518520 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:34.842380 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.842340 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n"] Nov 02 09:56:34.843041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843024 4467 cpu_manager.go:414] "containerMap: removing container" podUID="25989a4c-f1b8-47af-ae2b-bbbf87db2b8e" containerName="payload-pre-processing" Nov 02 09:56:34.843105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843042 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25989a4c-f1b8-47af-ae2b-bbbf87db2b8e" containerName="payload-pre-processing" Nov 02 09:56:34.843105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843058 4467 cpu_manager.go:414] "containerMap: removing container" podUID="6a96cc6a-e2b9-47fc-97ec-4829420592d1" containerName="maas-api" Nov 02 09:56:34.843105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843063 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a96cc6a-e2b9-47fc-97ec-4829420592d1" containerName="maas-api" Nov 02 09:56:34.843105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843071 4467 cpu_manager.go:414] "containerMap: removing container" podUID="d747555b-d2f7-4d7d-a2be-2828cfe813c5" containerName="payload-processing" Nov 02 09:56:34.843105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843076 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d747555b-d2f7-4d7d-a2be-2828cfe813c5" containerName="payload-processing" Nov 02 09:56:34.843105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843092 4467 cpu_manager.go:414] "containerMap: removing container" podUID="cdd86db9-3ec8-4db4-8a32-4053926cb17b" containerName="payload-processing" Nov 02 09:56:34.843105 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843097 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd86db9-3ec8-4db4-8a32-4053926cb17b" containerName="payload-processing" Nov 02 09:56:34.843327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843199 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="25989a4c-f1b8-47af-ae2b-bbbf87db2b8e" containerName="payload-pre-processing" Nov 02 09:56:34.843327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843207 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="d747555b-d2f7-4d7d-a2be-2828cfe813c5" containerName="payload-processing" Nov 02 09:56:34.843327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843214 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a96cc6a-e2b9-47fc-97ec-4829420592d1" containerName="maas-api" Nov 02 09:56:34.843327 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.843222 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd86db9-3ec8-4db4-8a32-4053926cb17b" containerName="payload-processing" Nov 02 09:56:34.848465 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.848441 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:34.855562 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.855533 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 09:56:34.857053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.857021 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dc8ee0e6e2cd\"" Nov 02 09:56:34.857053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.857020 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 09:56:34.857053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.857032 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 09:56:34.857290 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.857084 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-993442311390\"" Nov 02 09:56:34.857290 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.857110 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xdvq7\"" Nov 02 09:56:34.857499 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.857453 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 09:56:34.860837 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.860818 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n"] Nov 02 09:56:34.877647 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.877615 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-plugins-config-volume\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:34.877809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.877756 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-routing-overlay-volume\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:34.877938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.877879 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g7nw\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-kube-api-access-7g7nw\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:34.878037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.878018 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-provider-credentials\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:34.883522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:34.883489 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:35.006048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.006010 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-plugins-config-volume\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.007352 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.007328 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-routing-overlay-volume\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.015992 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.015960 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g7nw\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-kube-api-access-7g7nw\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.016287 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.016263 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-provider-credentials\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.035917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.007181 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-plugins-config-volume\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.035917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.008084 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-routing-overlay-volume\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.045919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.044575 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g7nw\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-kube-api-access-7g7nw\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.087916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.080722 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-provider-credentials\") pod \"payload-processing-external-model-f9c56bc9c-rph5n\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.167964 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.166173 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:35.529939 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.528162 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n"] Nov 02 09:56:35.544169 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:56:35.537081 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52136b2f_69bf_4443_87b4_bbc179575557.slice/crio-b2c05f04bfc8f48676bbf53f9d4cdf712b0d723ded8b7d03f2a2e6aee1b77315 WatchSource:0}: Error finding container b2c05f04bfc8f48676bbf53f9d4cdf712b0d723ded8b7d03f2a2e6aee1b77315: Status 404 returned error can't find the container with id b2c05f04bfc8f48676bbf53f9d4cdf712b0d723ded8b7d03f2a2e6aee1b77315 Nov 02 09:56:35.549689 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.549652 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:35.586874 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:35.586845 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" event={"ID":"52136b2f-69bf-4443-87b4-bbc179575557","Type":"ContainerStarted","Data":"b2c05f04bfc8f48676bbf53f9d4cdf712b0d723ded8b7d03f2a2e6aee1b77315"} Nov 02 09:56:36.345385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:36.345347 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:36.595620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:36.595520 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" event={"ID":"52136b2f-69bf-4443-87b4-bbc179575557","Type":"ContainerStarted","Data":"2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5"} Nov 02 09:56:36.595620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:36.595580 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:36.617200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:36.617136 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" podStartSLOduration=2.6171180019999998 podStartE2EDuration="2.617118002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:56:36.614516726 +0000 UTC m=+1275.454458717" watchObservedRunningTime="2026-11-02 09:56:36.617118002 +0000 UTC m=+1275.457059997" Nov 02 09:56:37.214147 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:37.214111 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:37.979853 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:37.979812 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:39.187778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:39.187715 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:39.956113 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:39.956081 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:40.639075 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:40.639011 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:41.316484 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:41.316451 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:42.004243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.004037 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:42.416721 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.416682 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:42.613047 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.613009 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:42.918778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.918747 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw"] Nov 02 09:56:42.922997 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.922947 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:42.931142 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.931117 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-9538968cbedc\"" Nov 02 09:56:42.933612 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.933584 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw"] Nov 02 09:56:42.973388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.971659 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vprg\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-kube-api-access-6vprg\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:42.973388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.971882 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:42.973388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.971992 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:42.973388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:42.972100 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.073406 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.073366 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.074029 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.073998 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.077243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.077183 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.077521 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.077492 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vprg\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-kube-api-access-6vprg\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.078038 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.074749 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.078171 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.074393 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.084861 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.084832 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.091209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.091181 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vprg\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-kube-api-access-6vprg\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.097067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.097036 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:43.239191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.239107 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.524058 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.524028 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw"] Nov 02 09:56:43.689844 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.689800 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" event={"ID":"0d93ff70-7684-45f4-b2ab-436559375028","Type":"ContainerStarted","Data":"92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c"} Nov 02 09:56:43.690044 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.689863 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" event={"ID":"0d93ff70-7684-45f4-b2ab-436559375028","Type":"ContainerStarted","Data":"535cd17bbeb28352ea0de06cf980f58f085320c6f358b2adfc0c01d79d7baeba"} Nov 02 09:56:43.690044 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.689906 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:43.711827 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.711775 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" podStartSLOduration=1.7117615640000001 podStartE2EDuration="1.711761564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:56:43.708264983 +0000 UTC m=+1282.548206978" watchObservedRunningTime="2026-11-02 09:56:43.711761564 +0000 UTC m=+1282.551703556" Nov 02 09:56:43.855813 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:43.855778 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:44.561845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:44.561806 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:45.446966 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:45.446930 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:46.034045 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:46.034000 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:46.222913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:46.222450 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:46.415047 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:46.414998 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:46.708631 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:46.708542 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:56:47.407918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:47.407652 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:47.599652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:47.599617 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:47.616911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:47.613148 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:56:48.461094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:48.461054 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:49.552514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:49.552457 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:50.678073 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:50.678030 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 09:56:50.678532 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:50.678092 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 09:56:51.069998 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:51.065438 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:52.670822 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:52.670789 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:53.579622 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:53.579582 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:54.170699 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:54.170657 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:54.563820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:54.563778 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:54.700625 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:54.700594 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:56:54.758488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:54.758454 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7"] Nov 02 09:56:54.878486 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:54.877110 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:55.555576 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:55.555515 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:56.269187 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:56.269149 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:56.855955 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:56:56.855920 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:56:56.973328 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:56.973293 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:57.777966 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:57.777929 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:58.862916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:58.862743 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:59.747605 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:59.747570 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:56:59.761593 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:56:59.761520 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" podUID="bdc2ab2e-9620-4a5f-be39-10b70a1775cd" containerName="payload-processing" containerID="cri-o://f156dd88934171584404279c2cc40618bb605dd195f71b3bcd7f434d66aeca22" gracePeriod=25 Nov 02 09:57:00.005143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.005045 4467 generic.go:356] "Generic (PLEG): container finished" podID="bdc2ab2e-9620-4a5f-be39-10b70a1775cd" containerID="f156dd88934171584404279c2cc40618bb605dd195f71b3bcd7f434d66aeca22" exitCode=0 Nov 02 09:57:00.005823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.005658 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" event={"ID":"bdc2ab2e-9620-4a5f-be39-10b70a1775cd","Type":"ContainerDied","Data":"f156dd88934171584404279c2cc40618bb605dd195f71b3bcd7f434d66aeca22"} Nov 02 09:57:00.702213 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.682305 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:57:00.812457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.812416 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-plugins-config-volume\") pod \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " Nov 02 09:57:00.813205 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.813180 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-kube-api-access-dpvxt\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-kube-api-access-dpvxt\") pod \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " Nov 02 09:57:00.813466 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.813444 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-provider-credentials\") pod \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " Nov 02 09:57:00.813773 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.813753 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-routing-overlay-volume\") pod \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\" (UID: \"bdc2ab2e-9620-4a5f-be39-10b70a1775cd\") " Nov 02 09:57:00.824029 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.813116 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-plugins-config-volume" pod "bdc2ab2e-9620-4a5f-be39-10b70a1775cd" (UID: "bdc2ab2e-9620-4a5f-be39-10b70a1775cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:00.824936 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.824911 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-routing-overlay-volume" pod "bdc2ab2e-9620-4a5f-be39-10b70a1775cd" (UID: "bdc2ab2e-9620-4a5f-be39-10b70a1775cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:00.843964 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.840121 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-provider-credentials" pod "bdc2ab2e-9620-4a5f-be39-10b70a1775cd" (UID: "bdc2ab2e-9620-4a5f-be39-10b70a1775cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:00.862915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.855475 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-kube-api-access-dpvxt" pod "bdc2ab2e-9620-4a5f-be39-10b70a1775cd" (UID: "bdc2ab2e-9620-4a5f-be39-10b70a1775cd"). InnerVolumeSpecName "kube-api-access-dpvxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:00.948748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.916937 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:00.948748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.916987 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:00.948748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.917005 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:00.948748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:00.917020 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvxt\" (UniqueName: \"kubernetes.io/projected/bdc2ab2e-9620-4a5f-be39-10b70a1775cd-kube-api-access-dpvxt\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:01.167109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:01.147168 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:01.207917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:01.203369 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" event={"ID":"bdc2ab2e-9620-4a5f-be39-10b70a1775cd","Type":"ContainerDied","Data":"3655990991b8b4d54e92620969bc0701c1a448fe119621eeec9c8f3e5996db0e"} Nov 02 09:57:01.207917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:01.203447 4467 scope.go:122] "RemoveContainer" containerID="f156dd88934171584404279c2cc40618bb605dd195f71b3bcd7f434d66aeca22" Nov 02 09:57:01.207917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:01.203617 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7" Nov 02 09:57:01.365153 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:01.365115 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:01.478059 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:01.477986 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7"] Nov 02 09:57:01.478280 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:01.478262 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76dgcj7"] Nov 02 09:57:01.566786 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:01.566747 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:01.677251 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:01.677175 4467 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc2ab2e_9620_4a5f_be39_10b70a1775cd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc2ab2e_9620_4a5f_be39_10b70a1775cd.slice/crio-3655990991b8b4d54e92620969bc0701c1a448fe119621eeec9c8f3e5996db0e\": RecentStats: unable to find data in memory cache]" Nov 02 09:57:02.078508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:02.078429 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:02.707034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:02.707000 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:03.161913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:03.159088 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:03.333832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:03.333790 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc2ab2e-9620-4a5f-be39-10b70a1775cd" path="/var/lib/kubelet/pods/bdc2ab2e-9620-4a5f-be39-10b70a1775cd/volumes" Nov 02 09:57:03.356633 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:03.356595 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:04.356407 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:04.356369 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:05.077825 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:05.077787 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:05.255612 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:05.255251 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:05.438946 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:05.438846 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:06.245595 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:06.245557 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n"] Nov 02 09:57:06.273394 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:06.273358 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:07.161216 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:07.161187 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:07.449097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:07.449009 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:08.257607 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:08.257574 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:08.334184 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:08.334149 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r"] Nov 02 09:57:08.334476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:08.334436 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" containerID="cri-o://ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048" gracePeriod=30 Nov 02 09:57:08.337322 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:08.337290 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t"] Nov 02 09:57:08.337564 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:08.337519 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" containerID="cri-o://d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73" gracePeriod=30 Nov 02 09:57:08.342243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:08.342221 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl"] Nov 02 09:57:08.342408 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:08.342391 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" containerID="cri-o://4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46" gracePeriod=30 Nov 02 09:57:09.341912 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:09.341552 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:09.682657 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:09.682576 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:57:10.461948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:10.459882 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:11.139729 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.139682 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:11.246447 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.246405 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" podUID="52136b2f-69bf-4443-87b4-bbc179575557" containerName="payload-processing" containerID="cri-o://2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5" gracePeriod=25 Nov 02 09:57:11.468421 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.468391 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:11.546618 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.546588 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:57:11.661625 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.661588 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-routing-overlay-volume\") pod \"52136b2f-69bf-4443-87b4-bbc179575557\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " Nov 02 09:57:11.661811 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.661788 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-kube-api-access-7g7nw\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-kube-api-access-7g7nw\") pod \"52136b2f-69bf-4443-87b4-bbc179575557\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " Nov 02 09:57:11.661953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.661934 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-provider-credentials\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-provider-credentials\") pod \"52136b2f-69bf-4443-87b4-bbc179575557\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " Nov 02 09:57:11.662044 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.662022 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-plugins-config-volume\") pod \"52136b2f-69bf-4443-87b4-bbc179575557\" (UID: \"52136b2f-69bf-4443-87b4-bbc179575557\") " Nov 02 09:57:11.662044 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.662032 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-routing-overlay-volume" pod "52136b2f-69bf-4443-87b4-bbc179575557" (UID: "52136b2f-69bf-4443-87b4-bbc179575557"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:11.662379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.662353 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-plugins-config-volume" pod "52136b2f-69bf-4443-87b4-bbc179575557" (UID: "52136b2f-69bf-4443-87b4-bbc179575557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:11.662575 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.662559 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:11.662646 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.662577 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52136b2f-69bf-4443-87b4-bbc179575557-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:11.664134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.664110 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-provider-credentials" pod "52136b2f-69bf-4443-87b4-bbc179575557" (UID: "52136b2f-69bf-4443-87b4-bbc179575557"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:11.664134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.664123 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-kube-api-access-7g7nw" pod "52136b2f-69bf-4443-87b4-bbc179575557" (UID: "52136b2f-69bf-4443-87b4-bbc179575557"). InnerVolumeSpecName "kube-api-access-7g7nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:11.763906 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.763813 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g7nw\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-kube-api-access-7g7nw\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:11.763906 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.763843 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52136b2f-69bf-4443-87b4-bbc179575557-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:11.937329 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:11.937243 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:12.344607 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.344573 4467 generic.go:356] "Generic (PLEG): container finished" podID="52136b2f-69bf-4443-87b4-bbc179575557" containerID="2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5" exitCode=0 Nov 02 09:57:12.344804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.344615 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" event={"ID":"52136b2f-69bf-4443-87b4-bbc179575557","Type":"ContainerDied","Data":"2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5"} Nov 02 09:57:12.344804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.344641 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" Nov 02 09:57:12.344804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.344680 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n" event={"ID":"52136b2f-69bf-4443-87b4-bbc179575557","Type":"ContainerDied","Data":"b2c05f04bfc8f48676bbf53f9d4cdf712b0d723ded8b7d03f2a2e6aee1b77315"} Nov 02 09:57:12.344804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.344710 4467 scope.go:122] "RemoveContainer" containerID="2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5" Nov 02 09:57:12.355678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.355654 4467 scope.go:122] "RemoveContainer" containerID="2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5" Nov 02 09:57:12.355994 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:12.355968 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5\": container with ID starting with 2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5 not found: ID does not exist" containerID="2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5" Nov 02 09:57:12.356117 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.356000 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5"} err="failed to get container status \"2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5\": rpc error: code = NotFound desc = could not find container \"2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5\": container with ID starting with 2e6ff00093134672ca27b03e768dc37cc58580635887ca2182a38c96d8f938c5 not found: ID does not exist" Nov 02 09:57:12.370328 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.370296 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n"] Nov 02 09:57:12.374880 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.374851 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f9c56bc9c-rph5n"] Nov 02 09:57:12.665746 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:12.665640 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:13.100378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.100346 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:13.333099 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.333068 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52136b2f-69bf-4443-87b4-bbc179575557" path="/var/lib/kubelet/pods/52136b2f-69bf-4443-87b4-bbc179575557/volumes" Nov 02 09:57:13.571314 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.569683 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:13.765270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.765121 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:13.882421 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.878094 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899169 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-kube-api-access-q6t2z\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-kube-api-access-q6t2z\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899222 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-envoy\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899260 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-credential-socket\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899334 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-data\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899415 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-socket\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899541 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8396fdd2-c412-459d-956d-e0a707278dd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8396fdd2-c412-459d-956d-e0a707278dd5-istiod-ca-cert\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899627 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-certs\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899683 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-istio-token\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-istio-token\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.899726 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8396fdd2-c412-459d-956d-e0a707278dd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8396fdd2-c412-459d-956d-e0a707278dd5-istio-podinfo\") pod \"8396fdd2-c412-459d-956d-e0a707278dd5\" (UID: \"8396fdd2-c412-459d-956d-e0a707278dd5\") " Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.906163 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8396fdd2-c412-459d-956d-e0a707278dd5-istio-podinfo" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.912495 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-kube-api-access-q6t2z" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "kube-api-access-q6t2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.913200 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8396fdd2-c412-459d-956d-e0a707278dd5-istiod-ca-cert" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.914412 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-credential-socket" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.915485 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-certs" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.915639 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-socket" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.916815 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-data" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.923439 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-istio-token" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:13.933535 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.931535 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-envoy" pod "8396fdd2-c412-459d-956d-e0a707278dd5" (UID: "8396fdd2-c412-459d-956d-e0a707278dd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:13.947090 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.947057 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:13.991275 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:13.990301 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:57:14.000829 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.000803 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-envoy\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.001108 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.000854 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-kube-api-access-8qkds\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-kube-api-access-8qkds\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.001315 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.001292 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-credential-socket\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.001405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.001353 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istiod-ca-cert\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.001470 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.001397 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-certs\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.001536 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.001464 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-data\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.001536 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.001503 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-podinfo\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.001652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.001594 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-token\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-token\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.001793 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.001774 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-socket\") pod \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\" (UID: \"07c5bc4b-f148-451e-b39e-bf9c717f37b2\") " Nov 02 09:57:14.002305 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002286 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8396fdd2-c412-459d-956d-e0a707278dd5-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002305 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002305 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002316 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002326 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8396fdd2-c412-459d-956d-e0a707278dd5-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002335 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6t2z\" (UniqueName: \"kubernetes.io/projected/8396fdd2-c412-459d-956d-e0a707278dd5-kube-api-access-q6t2z\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002345 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002353 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002362 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002371 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8396fdd2-c412-459d-956d-e0a707278dd5-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.002630 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002591 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-socket" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.002727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002712 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-credential-socket" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.002944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.002916 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istiod-ca-cert" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:14.003366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.003190 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-certs" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.005450 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.005428 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-data" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.005543 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.005511 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-envoy" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.005592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.005562 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-kube-api-access-8qkds" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "kube-api-access-8qkds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:14.005878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.005859 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-podinfo" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:57:14.007560 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.007532 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-token" pod "07c5bc4b-f148-451e-b39e-bf9c717f37b2" (UID: "07c5bc4b-f148-451e-b39e-bf9c717f37b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:14.015660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.015476 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:57:14.103665 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.103615 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-podinfo\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.103873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.103668 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-credential-socket\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.103873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.103712 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-kube-api-access-g7f76\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-kube-api-access-g7f76\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.103873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.103802 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-token\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-token\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.103873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.103855 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-data\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.104143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.103943 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-socket\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.104143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104013 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istiod-ca-cert\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.104143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104040 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-envoy\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.104143 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104127 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-certs\") pod \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\" (UID: \"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426\") " Nov 02 09:57:14.104538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104504 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-socket" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.104660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104560 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istiod-ca-cert" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:14.104842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104738 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-data" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.104842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104754 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-certs" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.104842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104818 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-credential-socket" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104923 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104943 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104955 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qkds\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-kube-api-access-8qkds\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104970 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104981 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104990 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.104999 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.105007 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.105015 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c5bc4b-f148-451e-b39e-bf9c717f37b2-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.105024 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.105222 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.105034 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.106659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.106632 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-kube-api-access-g7f76" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "kube-api-access-g7f76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:14.107196 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.107164 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-token" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:14.107279 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.107218 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-podinfo" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:57:14.108459 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.108433 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-envoy" pod "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" (UID: "6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:57:14.138653 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.137119 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:14.210758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.210719 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.210758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.210757 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.211094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.210776 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7f76\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-kube-api-access-g7f76\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.211094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.210787 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.211094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.210796 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.211094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.210806 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.211094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.210819 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:14.314547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.314509 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:14.362018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.361981 4467 generic.go:356] "Generic (PLEG): container finished" podID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerID="4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46" exitCode=0 Nov 02 09:57:14.362208 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.362034 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" Nov 02 09:57:14.362208 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.362059 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" event={"ID":"07c5bc4b-f148-451e-b39e-bf9c717f37b2","Type":"ContainerDied","Data":"4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46"} Nov 02 09:57:14.362208 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.362117 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl" event={"ID":"07c5bc4b-f148-451e-b39e-bf9c717f37b2","Type":"ContainerDied","Data":"2beea59ba136adac0b95b8f627a8239ee941d7b0787880c84b9ffd9ff819723c"} Nov 02 09:57:14.362208 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.362151 4467 scope.go:122] "RemoveContainer" containerID="4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46" Nov 02 09:57:14.363606 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.363582 4467 generic.go:356] "Generic (PLEG): container finished" podID="8396fdd2-c412-459d-956d-e0a707278dd5" containerID="ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048" exitCode=0 Nov 02 09:57:14.363727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.363639 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" Nov 02 09:57:14.363727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.363635 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" event={"ID":"8396fdd2-c412-459d-956d-e0a707278dd5","Type":"ContainerDied","Data":"ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048"} Nov 02 09:57:14.363938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.363739 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r" event={"ID":"8396fdd2-c412-459d-956d-e0a707278dd5","Type":"ContainerDied","Data":"c460b4133f24227116aa481c29f3c03f666733df7f03558194f7e2d411e4e78e"} Nov 02 09:57:14.365517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.365371 4467 generic.go:356] "Generic (PLEG): container finished" podID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerID="d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73" exitCode=0 Nov 02 09:57:14.365517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.365420 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" Nov 02 09:57:14.365517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.365451 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" event={"ID":"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426","Type":"ContainerDied","Data":"d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73"} Nov 02 09:57:14.365517 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.365488 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t" event={"ID":"6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426","Type":"ContainerDied","Data":"602014c2680aaf401ff27662f5117ea340db7666491029d00314f87941f0d990"} Nov 02 09:57:14.399324 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.399297 4467 scope.go:122] "RemoveContainer" containerID="4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46" Nov 02 09:57:14.399704 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:14.399685 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46\": container with ID starting with 4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46 not found: ID does not exist" containerID="4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46" Nov 02 09:57:14.399778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.399713 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46"} err="failed to get container status \"4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46\": rpc error: code = NotFound desc = could not find container \"4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46\": container with ID starting with 4ca878527bc893dbe3996bf211ff7193b775482af782487b7551e07cbd368f46 not found: ID does not exist" Nov 02 09:57:14.399778 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.399731 4467 scope.go:122] "RemoveContainer" containerID="ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048" Nov 02 09:57:14.409122 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.409104 4467 scope.go:122] "RemoveContainer" containerID="ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048" Nov 02 09:57:14.409426 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:14.409408 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048\": container with ID starting with ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048 not found: ID does not exist" containerID="ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048" Nov 02 09:57:14.409485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.409433 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048"} err="failed to get container status \"ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048\": rpc error: code = NotFound desc = could not find container \"ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048\": container with ID starting with ad3b7d4e85ecdac742fe3a9956f34e7b9872ee118c3c0b8578226050af36a048 not found: ID does not exist" Nov 02 09:57:14.409485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.409453 4467 scope.go:122] "RemoveContainer" containerID="d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73" Nov 02 09:57:14.418534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.418512 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl"] Nov 02 09:57:14.421800 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.421777 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p2lgl"] Nov 02 09:57:14.423036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.423014 4467 scope.go:122] "RemoveContainer" containerID="d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73" Nov 02 09:57:14.423342 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:14.423320 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73\": container with ID starting with d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73 not found: ID does not exist" containerID="d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73" Nov 02 09:57:14.423417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.423354 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73"} err="failed to get container status \"d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73\": rpc error: code = NotFound desc = could not find container \"d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73\": container with ID starting with d10d2a262add9f73bb18db87b25ba4a2fc6cbd32be4998b6c727022dce1a5b73 not found: ID does not exist" Nov 02 09:57:14.432410 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.432383 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r"] Nov 02 09:57:14.435201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.435177 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-s2h9r"] Nov 02 09:57:14.448129 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.448103 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t"] Nov 02 09:57:14.451128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.451103 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-qkk8t"] Nov 02 09:57:14.656435 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:14.656327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:15.333386 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.333352 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" path="/var/lib/kubelet/pods/07c5bc4b-f148-451e-b39e-bf9c717f37b2/volumes" Nov 02 09:57:15.333818 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.333803 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" path="/var/lib/kubelet/pods/6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426/volumes" Nov 02 09:57:15.334244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.334230 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" path="/var/lib/kubelet/pods/8396fdd2-c412-459d-956d-e0a707278dd5/volumes" Nov 02 09:57:15.485244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.485206 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:15.687854 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.687771 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:15.968558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.968465 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t"] Nov 02 09:57:15.969054 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969038 4467 cpu_manager.go:414] "containerMap: removing container" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" Nov 02 09:57:15.969054 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969054 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" Nov 02 09:57:15.969226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969069 4467 cpu_manager.go:414] "containerMap: removing container" podUID="52136b2f-69bf-4443-87b4-bbc179575557" containerName="payload-processing" Nov 02 09:57:15.969226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969075 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52136b2f-69bf-4443-87b4-bbc179575557" containerName="payload-processing" Nov 02 09:57:15.969226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969082 4467 cpu_manager.go:414] "containerMap: removing container" podUID="bdc2ab2e-9620-4a5f-be39-10b70a1775cd" containerName="payload-processing" Nov 02 09:57:15.969226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969088 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc2ab2e-9620-4a5f-be39-10b70a1775cd" containerName="payload-processing" Nov 02 09:57:15.969226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969134 4467 cpu_manager.go:414] "containerMap: removing container" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" Nov 02 09:57:15.969226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969139 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" Nov 02 09:57:15.969226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969150 4467 cpu_manager.go:414] "containerMap: removing container" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" Nov 02 09:57:15.969226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969155 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" Nov 02 09:57:15.969603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969241 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="52136b2f-69bf-4443-87b4-bbc179575557" containerName="payload-processing" Nov 02 09:57:15.969603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969251 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="8396fdd2-c412-459d-956d-e0a707278dd5" containerName="istio-proxy" Nov 02 09:57:15.969603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969257 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c5bc4b-f148-451e-b39e-bf9c717f37b2" containerName="istio-proxy" Nov 02 09:57:15.969603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969264 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ada5f7d-1d30-4bc3-b1ac-9d700d1e4426" containerName="istio-proxy" Nov 02 09:57:15.969603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.969270 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc2ab2e-9620-4a5f-be39-10b70a1775cd" containerName="payload-processing" Nov 02 09:57:15.974460 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.974441 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:15.985334 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.983215 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-0381f66eae89\"" Nov 02 09:57:15.985688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:15.985665 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t"] Nov 02 09:57:16.045700 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.045660 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.045909 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.045747 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2hh\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-kube-api-access-kl2hh\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.045985 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.045940 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.046042 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.045996 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.147485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.147441 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.147664 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.147519 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.147783 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.147760 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.147842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.147826 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2hh\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-kube-api-access-kl2hh\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.148416 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.148386 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.148593 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.148542 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.150547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.150525 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.162080 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.162051 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2hh\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-kube-api-access-kl2hh\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.291342 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.291243 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:16.371185 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.370588 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:16.573917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.571218 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:16.636904 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.636328 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t"] Nov 02 09:57:16.754952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.754773 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:16.977986 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:16.977865 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t"] Nov 02 09:57:17.030257 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.030227 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc"] Nov 02 09:57:17.034351 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.034329 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.039452 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.039430 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-1881b2c46ba8\"" Nov 02 09:57:17.049439 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.049414 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc"] Nov 02 09:57:17.095585 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.095550 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.095585 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.095591 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.095845 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.095796 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.095920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.095862 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhs8\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-kube-api-access-rwhs8\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.196981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.196935 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhs8\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-kube-api-access-rwhs8\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.197180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.197091 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.197180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.197117 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.197308 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.197288 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.197950 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.197884 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.198090 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.198068 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.200048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.200024 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.211376 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.211347 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhs8\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-kube-api-access-rwhs8\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.349489 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.349452 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:17.392049 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.392009 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" event={"ID":"59513e78-5732-47c1-bc2b-b424ec2d8443","Type":"ContainerStarted","Data":"28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2"} Nov 02 09:57:17.392550 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.392083 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:17.392550 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.392109 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" event={"ID":"59513e78-5732-47c1-bc2b-b424ec2d8443","Type":"ContainerStarted","Data":"e78fbf9729993a95dc0a451f36fb9647c64c2927fa4257ed23e519a46f126400"} Nov 02 09:57:17.420331 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.418863 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" podStartSLOduration=2.418837291 podStartE2EDuration="2.418837291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:17.414729603 +0000 UTC m=+1316.254671615" watchObservedRunningTime="2026-11-02 09:57:17.418837291 +0000 UTC m=+1316.258779288" Nov 02 09:57:17.636528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.636465 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc"] Nov 02 09:57:17.638995 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:57:17.638946 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f1ed85_8366_42bd_87a1_97fc1cf92825.slice/crio-c24cfefdd1456d2691a36ada9ea58391d8dfc7787f7a36d867b74907b0bbf47e WatchSource:0}: Error finding container c24cfefdd1456d2691a36ada9ea58391d8dfc7787f7a36d867b74907b0bbf47e: Status 404 returned error can't find the container with id c24cfefdd1456d2691a36ada9ea58391d8dfc7787f7a36d867b74907b0bbf47e Nov 02 09:57:17.960864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:17.960770 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:18.401996 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:18.401954 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" event={"ID":"22f1ed85-8366-42bd-87a1-97fc1cf92825","Type":"ContainerStarted","Data":"94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813"} Nov 02 09:57:18.402497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:18.402186 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:18.402497 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:18.402226 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" event={"ID":"22f1ed85-8366-42bd-87a1-97fc1cf92825","Type":"ContainerStarted","Data":"c24cfefdd1456d2691a36ada9ea58391d8dfc7787f7a36d867b74907b0bbf47e"} Nov 02 09:57:18.443273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:18.443219 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" podStartSLOduration=2.443203295 podStartE2EDuration="2.443203295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:18.441457804 +0000 UTC m=+1317.281399800" watchObservedRunningTime="2026-11-02 09:57:18.443203295 +0000 UTC m=+1317.283145289" Nov 02 09:57:18.838505 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:18.838472 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:19.314354 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:19.314318 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:19.562104 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:19.562016 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:20.273096 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:20.273024 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:21.447915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:21.442611 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:22.474363 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:22.474263 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:23.405189 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:23.404976 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" podUID="59513e78-5732-47c1-bc2b-b424ec2d8443" containerName="payload-processing" containerID="cri-o://28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2" gracePeriod=25 Nov 02 09:57:23.543917 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:23.540023 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:24.505914 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.497330 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:24.538915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.523914 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-routing-overlay-volume\") pod \"59513e78-5732-47c1-bc2b-b424ec2d8443\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " Nov 02 09:57:24.538915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.538080 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-plugins-config-volume\") pod \"59513e78-5732-47c1-bc2b-b424ec2d8443\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " Nov 02 09:57:24.538915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.538219 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-provider-credentials\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-provider-credentials\") pod \"59513e78-5732-47c1-bc2b-b424ec2d8443\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " Nov 02 09:57:24.538915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.538316 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-kube-api-access-kl2hh\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-kube-api-access-kl2hh\") pod \"59513e78-5732-47c1-bc2b-b424ec2d8443\" (UID: \"59513e78-5732-47c1-bc2b-b424ec2d8443\") " Nov 02 09:57:24.570596 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.570558 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-routing-overlay-volume" pod "59513e78-5732-47c1-bc2b-b424ec2d8443" (UID: "59513e78-5732-47c1-bc2b-b424ec2d8443"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:24.571179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.570656 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-plugins-config-volume" pod "59513e78-5732-47c1-bc2b-b424ec2d8443" (UID: "59513e78-5732-47c1-bc2b-b424ec2d8443"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:24.576788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.576706 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-kube-api-access-kl2hh" pod "59513e78-5732-47c1-bc2b-b424ec2d8443" (UID: "59513e78-5732-47c1-bc2b-b424ec2d8443"). InnerVolumeSpecName "kube-api-access-kl2hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:24.577469 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.577439 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-provider-credentials" pod "59513e78-5732-47c1-bc2b-b424ec2d8443" (UID: "59513e78-5732-47c1-bc2b-b424ec2d8443"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:24.681690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.681639 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:24.681690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.681688 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:24.682077 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.681703 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl2hh\" (UniqueName: \"kubernetes.io/projected/59513e78-5732-47c1-bc2b-b424ec2d8443-kube-api-access-kl2hh\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:24.682077 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.681717 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59513e78-5732-47c1-bc2b-b424ec2d8443-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:24.750029 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.749975 4467 generic.go:356] "Generic (PLEG): container finished" podID="59513e78-5732-47c1-bc2b-b424ec2d8443" containerID="28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2" exitCode=0 Nov 02 09:57:24.750351 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.750323 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" event={"ID":"59513e78-5732-47c1-bc2b-b424ec2d8443","Type":"ContainerDied","Data":"28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2"} Nov 02 09:57:24.753915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.750518 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" event={"ID":"59513e78-5732-47c1-bc2b-b424ec2d8443","Type":"ContainerDied","Data":"e78fbf9729993a95dc0a451f36fb9647c64c2927fa4257ed23e519a46f126400"} Nov 02 09:57:24.784916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.751050 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t" Nov 02 09:57:24.795595 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.751069 4467 scope.go:122] "RemoveContainer" containerID="28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2" Nov 02 09:57:24.801583 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.801552 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:24.915520 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.912077 4467 scope.go:122] "RemoveContainer" containerID="28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2" Nov 02 09:57:24.915520 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:24.914302 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2\": container with ID starting with 28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2 not found: ID does not exist" containerID="28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2" Nov 02 09:57:24.915520 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.914339 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2"} err="failed to get container status \"28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2\": rpc error: code = NotFound desc = could not find container \"28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2\": container with ID starting with 28e2469765386b99d1cf7628208c291a60bad6a64e2937d7e09cf39ee8e2a2a2 not found: ID does not exist" Nov 02 09:57:24.919997 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.919967 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t"] Nov 02 09:57:24.924443 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:24.924417 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bwxz4t"] Nov 02 09:57:25.334606 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:25.334575 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59513e78-5732-47c1-bc2b-b424ec2d8443" path="/var/lib/kubelet/pods/59513e78-5732-47c1-bc2b-b424ec2d8443/volumes" Nov 02 09:57:26.098490 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:26.098406 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:26.857991 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:26.857953 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:57:26.937794 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:26.937746 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:27.611070 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:27.611024 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:28.351072 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:28.351032 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:29.001938 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:29.001885 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:57:29.170629 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:29.170599 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:29.413600 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:29.413572 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:57:29.471730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:29.471693 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw"] Nov 02 09:57:29.870823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:29.870783 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:30.065533 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:30.065500 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:30.573405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:30.573357 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:31.364289 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:31.364252 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:32.070067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.068966 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:32.504388 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.504297 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:32.695948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.695912 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:32.989425 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.989390 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv"] Nov 02 09:57:32.990098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.990077 4467 cpu_manager.go:414] "containerMap: removing container" podUID="59513e78-5732-47c1-bc2b-b424ec2d8443" containerName="payload-processing" Nov 02 09:57:32.990098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.990098 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59513e78-5732-47c1-bc2b-b424ec2d8443" containerName="payload-processing" Nov 02 09:57:32.990297 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.990284 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="59513e78-5732-47c1-bc2b-b424ec2d8443" containerName="payload-processing" Nov 02 09:57:32.993745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.993725 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:32.998758 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:32.998714 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 09:57:33.000026 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.000002 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 09:57:33.000145 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.000031 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dc8ee0e6e2cd\"" Nov 02 09:57:33.000145 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.000068 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 09:57:33.000145 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.000110 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xdvq7\"" Nov 02 09:57:33.000145 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.000116 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-558c3c185b7f\"" Nov 02 09:57:33.006066 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.006044 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 09:57:33.006189 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.006141 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d55c41d94c02\"" Nov 02 09:57:33.006430 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.006409 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv"] Nov 02 09:57:33.104258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.104216 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-routing-overlay-volume\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.104446 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.104283 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-plugins-config-volume\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.104446 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.104360 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt78b\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-kube-api-access-wt78b\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.104528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.104496 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-provider-credentials\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.205660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.205621 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-routing-overlay-volume\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.205860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.205676 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-plugins-config-volume\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.205860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.205705 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt78b\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-kube-api-access-wt78b\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.205860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.205774 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-provider-credentials\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.206508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.206476 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-routing-overlay-volume\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.206636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.206557 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-plugins-config-volume\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.208438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.208417 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-provider-credentials\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.218168 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.218142 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt78b\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-kube-api-access-wt78b\") pod \"payload-processing-external-model-5ff7768868-lfrbv\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.311508 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.311479 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.493739 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.493707 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:33.596614 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.596582 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv"] Nov 02 09:57:33.597056 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:57:33.597033 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0977d0_2ce6_4177_b59d_966560321782.slice/crio-ec5449249693c58188c638206987bbbddeb3de22787fe1a861d1f138956c9661 WatchSource:0}: Error finding container ec5449249693c58188c638206987bbbddeb3de22787fe1a861d1f138956c9661: Status 404 returned error can't find the container with id ec5449249693c58188c638206987bbbddeb3de22787fe1a861d1f138956c9661 Nov 02 09:57:33.824932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.824591 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" event={"ID":"ba0977d0-2ce6-4177-b59d-966560321782","Type":"ContainerStarted","Data":"f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75"} Nov 02 09:57:33.824932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.824693 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" event={"ID":"ba0977d0-2ce6-4177-b59d-966560321782","Type":"ContainerStarted","Data":"ec5449249693c58188c638206987bbbddeb3de22787fe1a861d1f138956c9661"} Nov 02 09:57:33.824932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.824847 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:33.846720 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:33.846629 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" podStartSLOduration=1.846614959 podStartE2EDuration="1.846614959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:33.845039607 +0000 UTC m=+1332.684981601" watchObservedRunningTime="2026-11-02 09:57:33.846614959 +0000 UTC m=+1332.686556952" Nov 02 09:57:34.251048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.250945 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:34.472389 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.472344 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" podUID="0d93ff70-7684-45f4-b2ab-436559375028" containerName="payload-processing" containerID="cri-o://92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c" gracePeriod=25 Nov 02 09:57:34.755532 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.755468 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:57:34.833817 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.833785 4467 generic.go:356] "Generic (PLEG): container finished" podID="0d93ff70-7684-45f4-b2ab-436559375028" containerID="92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c" exitCode=0 Nov 02 09:57:34.834006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.833830 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" Nov 02 09:57:34.834006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.833875 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" event={"ID":"0d93ff70-7684-45f4-b2ab-436559375028","Type":"ContainerDied","Data":"92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c"} Nov 02 09:57:34.834006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.833965 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" event={"ID":"0d93ff70-7684-45f4-b2ab-436559375028","Type":"ContainerDied","Data":"535cd17bbeb28352ea0de06cf980f58f085320c6f358b2adfc0c01d79d7baeba"} Nov 02 09:57:34.834175 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.834006 4467 scope.go:122] "RemoveContainer" containerID="92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c" Nov 02 09:57:34.843138 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.843108 4467 scope.go:122] "RemoveContainer" containerID="92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c" Nov 02 09:57:34.843431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.843411 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-provider-credentials\") pod \"0d93ff70-7684-45f4-b2ab-436559375028\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " Nov 02 09:57:34.843543 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:34.843410 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c\": container with ID starting with 92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c not found: ID does not exist" containerID="92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c" Nov 02 09:57:34.843609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.843563 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c"} err="failed to get container status \"92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c\": rpc error: code = NotFound desc = could not find container \"92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c\": container with ID starting with 92b1ef5c85bb312e275c5d775c678af43a0b96699e0178b9c8475d5172aa4d7c not found: ID does not exist" Nov 02 09:57:34.843609 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.843586 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-plugins-config-volume\") pod \"0d93ff70-7684-45f4-b2ab-436559375028\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " Nov 02 09:57:34.843733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.843617 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-kube-api-access-6vprg\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-kube-api-access-6vprg\") pod \"0d93ff70-7684-45f4-b2ab-436559375028\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " Nov 02 09:57:34.843733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.843644 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-routing-overlay-volume\") pod \"0d93ff70-7684-45f4-b2ab-436559375028\" (UID: \"0d93ff70-7684-45f4-b2ab-436559375028\") " Nov 02 09:57:34.844076 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.844052 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-routing-overlay-volume" pod "0d93ff70-7684-45f4-b2ab-436559375028" (UID: "0d93ff70-7684-45f4-b2ab-436559375028"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:34.844076 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.844061 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-plugins-config-volume" pod "0d93ff70-7684-45f4-b2ab-436559375028" (UID: "0d93ff70-7684-45f4-b2ab-436559375028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:57:34.844247 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.844222 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:34.844247 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.844243 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d93ff70-7684-45f4-b2ab-436559375028-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:34.845718 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.845692 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-kube-api-access-6vprg" pod "0d93ff70-7684-45f4-b2ab-436559375028" (UID: "0d93ff70-7684-45f4-b2ab-436559375028"). InnerVolumeSpecName "kube-api-access-6vprg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:34.845808 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.845759 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-provider-credentials" pod "0d93ff70-7684-45f4-b2ab-436559375028" (UID: "0d93ff70-7684-45f4-b2ab-436559375028"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:57:34.942085 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.942026 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:34.946696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.946666 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vprg\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-kube-api-access-6vprg\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:34.946696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:34.946696 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d93ff70-7684-45f4-b2ab-436559375028-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:57:35.158506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:35.158462 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw"] Nov 02 09:57:35.168414 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:35.168380 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw"] Nov 02 09:57:35.334493 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:35.334456 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d93ff70-7684-45f4-b2ab-436559375028" path="/var/lib/kubelet/pods/0d93ff70-7684-45f4-b2ab-436559375028/volumes" Nov 02 09:57:35.698764 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:35.698711 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dfkrbw" podUID="0d93ff70-7684-45f4-b2ab-436559375028" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.86:50052\" within 1s: context deadline exceeded" Nov 02 09:57:35.699022 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:57:35.698865 4467 logging.go:55] [core] [Channel #593 SubChannel #594]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.86:50052", ServerName: "10.133.0.86:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.86:50052: operation was canceled" Nov 02 09:57:35.979171 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:35.979051 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:36.785838 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:36.785802 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:37.136913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:37.136429 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:37.909686 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:37.909529 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:39.377775 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:39.377704 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:40.363711 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:40.363677 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:41.107975 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:41.107933 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:41.420381 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:41.420290 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:57:41.745588 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:41.745490 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:43.008235 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:43.008199 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:43.866674 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:43.866634 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:44.861857 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:44.861149 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:57:45.322270 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:45.322227 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:46.616916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:46.614136 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:47.740502 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:47.740446 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:48.445134 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:48.445099 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:49.156681 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:49.156641 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:50.151733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:50.151685 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:51.107915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:51.103519 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:52.396799 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:52.396700 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:54.133119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:54.133065 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:55.031315 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:55.030575 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:55.587897 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:55.587849 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:57:55.881329 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:55.881234 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:56.727745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:56.727710 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:56.856420 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:57:56.856387 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:57:57.320727 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:57.320689 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:57.505209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:57.505164 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:58.441581 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:58.441542 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:58.668002 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:58.667968 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:58.850683 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:58.850645 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:59.122311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.122217 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg"] Nov 02 09:57:59.123037 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.123008 4467 cpu_manager.go:414] "containerMap: removing container" podUID="0d93ff70-7684-45f4-b2ab-436559375028" containerName="payload-processing" Nov 02 09:57:59.123150 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.123040 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d93ff70-7684-45f4-b2ab-436559375028" containerName="payload-processing" Nov 02 09:57:59.123280 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.123215 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d93ff70-7684-45f4-b2ab-436559375028" containerName="payload-processing" Nov 02 09:57:59.128135 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.128114 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.136018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.135992 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-a017d0242320\"" Nov 02 09:57:59.140483 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.140451 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg"] Nov 02 09:57:59.191499 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.191464 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq7cj\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-kube-api-access-wq7cj\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.191714 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.191502 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.191714 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.191702 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.191790 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.191719 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.293056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.293019 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq7cj\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-kube-api-access-wq7cj\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.293056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.293066 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.293320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.293249 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.293320 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.293266 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.293811 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.293783 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.294124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.294098 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.295989 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.295967 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.305797 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.305770 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq7cj\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-kube-api-access-wq7cj\") pod \"payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.443420 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.443331 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:57:59.651495 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.651313 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:57:59.739455 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:57:59.739420 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c134a0_dfe1_4586_b5cc_86eb9b4dafe1.slice/crio-a352f6a693beecd2f1d29e098b1ae47aaffae5dc223e422a87b54e3ff484255b WatchSource:0}: Error finding container a352f6a693beecd2f1d29e098b1ae47aaffae5dc223e422a87b54e3ff484255b: Status 404 returned error can't find the container with id a352f6a693beecd2f1d29e098b1ae47aaffae5dc223e422a87b54e3ff484255b Nov 02 09:57:59.743259 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:57:59.743234 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg"] Nov 02 09:58:00.331962 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:00.330074 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" event={"ID":"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1","Type":"ContainerStarted","Data":"2f5138acf45d45c3eb2eab4e8b217ce683d0d16ac1f7dbc86024a8622a202570"} Nov 02 09:58:00.331962 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:00.330322 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" event={"ID":"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1","Type":"ContainerStarted","Data":"a352f6a693beecd2f1d29e098b1ae47aaffae5dc223e422a87b54e3ff484255b"} Nov 02 09:58:00.331962 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:00.331480 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:58:00.346468 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:00.346425 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:00.353881 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:00.353810 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" podStartSLOduration=1.353789626 podStartE2EDuration="1.353789626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:00.350126405 +0000 UTC m=+1359.190068400" watchObservedRunningTime="2026-11-02 09:58:00.353789626 +0000 UTC m=+1359.193731621" Nov 02 09:58:01.156041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:01.156008 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:01.861628 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:01.861593 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:02.543350 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:02.543317 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:02.725606 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:02.725573 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:03.038657 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:03.038624 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:03.374824 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:03.374788 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:04.158740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:04.158694 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:04.941040 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:04.941007 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:05.854826 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:05.854788 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:06.554689 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:06.552695 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:07.350570 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:07.350539 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:08.092626 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:08.092593 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:08.758922 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:08.758862 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:08.941668 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:08.941631 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:09.113991 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.113954 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk"] Nov 02 09:58:09.124144 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.124101 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.157911 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.144447 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk"] Nov 02 09:58:09.158472 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.148130 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:09.262454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.260124 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-provider-credentials\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.262454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.262043 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-routing-overlay-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.262454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.262080 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgrc2\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-kube-api-access-zgrc2\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.262454 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.262258 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-plugins-config-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.363713 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.363665 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-provider-credentials\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.363961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.363729 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-routing-overlay-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.363961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.363758 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgrc2\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-kube-api-access-zgrc2\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.364138 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.363969 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-plugins-config-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.365043 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.365014 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-plugins-config-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.365688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.365659 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-routing-overlay-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.367476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.367454 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-provider-credentials\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.377163 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.377121 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgrc2\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-kube-api-access-zgrc2\") pod \"payload-processing-external-model-7b5bbbbbb8-ndszk\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.505062 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.505025 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:09.907849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:09.907804 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk"] Nov 02 09:58:09.908396 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:58:09.908368 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9417297f_ed2e_4748_8945_2a391a9809bb.slice/crio-c6cfaba82bcef3c14f995909a9ef6c26a328e4dfe9cf4efc46ffa07bc314e952 WatchSource:0}: Error finding container c6cfaba82bcef3c14f995909a9ef6c26a328e4dfe9cf4efc46ffa07bc314e952: Status 404 returned error can't find the container with id c6cfaba82bcef3c14f995909a9ef6c26a328e4dfe9cf4efc46ffa07bc314e952 Nov 02 09:58:10.404091 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:10.404054 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" event={"ID":"9417297f-ed2e-4748-8945-2a391a9809bb","Type":"ContainerStarted","Data":"a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8"} Nov 02 09:58:10.404319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:10.404113 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" event={"ID":"9417297f-ed2e-4748-8945-2a391a9809bb","Type":"ContainerStarted","Data":"c6cfaba82bcef3c14f995909a9ef6c26a328e4dfe9cf4efc46ffa07bc314e952"} Nov 02 09:58:10.404319 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:10.404289 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:10.416007 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:10.415970 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:10.427479 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:10.427417 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" podStartSLOduration=1.427397122 podStartE2EDuration="1.427397122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:10.424377236 +0000 UTC m=+1369.264319242" watchObservedRunningTime="2026-11-02 09:58:10.427397122 +0000 UTC m=+1369.267339118" Nov 02 09:58:10.663246 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:10.663162 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:11.147411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:11.147370 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:11.527549 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:11.527459 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:12.086403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:12.086355 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:58:12.087355 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:12.087328 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:12.344642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:12.344562 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:58:12.417989 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:12.417953 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc"] Nov 02 09:58:12.753648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:12.753564 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:13.902854 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:13.902813 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:15.093379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:15.093343 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:16.134934 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:16.134014 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:16.939940 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:16.937130 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:17.160322 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.160280 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:17.418452 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.418409 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" podUID="22f1ed85-8366-42bd-87a1-97fc1cf92825" containerName="payload-processing" containerID="cri-o://94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813" gracePeriod=25 Nov 02 09:58:17.674966 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.674837 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:17.734790 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.734493 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:58:17.832683 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.832650 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-plugins-config-volume\") pod \"22f1ed85-8366-42bd-87a1-97fc1cf92825\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " Nov 02 09:58:17.833387 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.833349 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-routing-overlay-volume\") pod \"22f1ed85-8366-42bd-87a1-97fc1cf92825\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " Nov 02 09:58:17.833529 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.833286 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-plugins-config-volume" pod "22f1ed85-8366-42bd-87a1-97fc1cf92825" (UID: "22f1ed85-8366-42bd-87a1-97fc1cf92825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:17.833629 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.833609 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-routing-overlay-volume" pod "22f1ed85-8366-42bd-87a1-97fc1cf92825" (UID: "22f1ed85-8366-42bd-87a1-97fc1cf92825"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:17.833690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.833628 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-provider-credentials\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-provider-credentials\") pod \"22f1ed85-8366-42bd-87a1-97fc1cf92825\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " Nov 02 09:58:17.833740 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.833718 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-kube-api-access-rwhs8\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-kube-api-access-rwhs8\") pod \"22f1ed85-8366-42bd-87a1-97fc1cf92825\" (UID: \"22f1ed85-8366-42bd-87a1-97fc1cf92825\") " Nov 02 09:58:17.834494 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.834469 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:17.834494 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.834495 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22f1ed85-8366-42bd-87a1-97fc1cf92825-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:17.836570 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.836543 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-provider-credentials" pod "22f1ed85-8366-42bd-87a1-97fc1cf92825" (UID: "22f1ed85-8366-42bd-87a1-97fc1cf92825"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:17.836702 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.836642 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-kube-api-access-rwhs8" pod "22f1ed85-8366-42bd-87a1-97fc1cf92825" (UID: "22f1ed85-8366-42bd-87a1-97fc1cf92825"). InnerVolumeSpecName "kube-api-access-rwhs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:17.935861 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.935784 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:17.935861 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:17.935814 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwhs8\" (UniqueName: \"kubernetes.io/projected/22f1ed85-8366-42bd-87a1-97fc1cf92825-kube-api-access-rwhs8\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:18.479754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.479705 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:18.484720 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.484685 4467 generic.go:356] "Generic (PLEG): container finished" podID="22f1ed85-8366-42bd-87a1-97fc1cf92825" containerID="94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813" exitCode=0 Nov 02 09:58:18.484878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.484747 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" Nov 02 09:58:18.484878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.484782 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" event={"ID":"22f1ed85-8366-42bd-87a1-97fc1cf92825","Type":"ContainerDied","Data":"94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813"} Nov 02 09:58:18.484878 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.484854 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc" event={"ID":"22f1ed85-8366-42bd-87a1-97fc1cf92825","Type":"ContainerDied","Data":"c24cfefdd1456d2691a36ada9ea58391d8dfc7787f7a36d867b74907b0bbf47e"} Nov 02 09:58:18.485086 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.484885 4467 scope.go:122] "RemoveContainer" containerID="94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813" Nov 02 09:58:18.496932 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.496908 4467 scope.go:122] "RemoveContainer" containerID="94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813" Nov 02 09:58:18.497225 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:18.497205 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813\": container with ID starting with 94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813 not found: ID does not exist" containerID="94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813" Nov 02 09:58:18.497296 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.497234 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813"} err="failed to get container status \"94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813\": rpc error: code = NotFound desc = could not find container \"94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813\": container with ID starting with 94b370881d8c8006af4a83cfc9be56ec2da075c10c4a265324884b552c9c4813 not found: ID does not exist" Nov 02 09:58:18.515649 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.515049 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc"] Nov 02 09:58:18.518584 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:18.518559 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55xrnzc"] Nov 02 09:58:19.252119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:19.251487 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:19.332110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:19.332076 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f1ed85-8366-42bd-87a1-97fc1cf92825" path="/var/lib/kubelet/pods/22f1ed85-8366-42bd-87a1-97fc1cf92825/volumes" Nov 02 09:58:19.939990 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:19.939959 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:20.657931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:20.657882 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:20.953372 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:20.953285 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk"] Nov 02 09:58:20.958032 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:20.958007 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:20.958191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:20.958061 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv"] Nov 02 09:58:21.063308 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:21.063276 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:21.275061 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:21.270684 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:21.435695 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:21.435660 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq"] Nov 02 09:58:21.435993 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:21.435937 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" podUID="62503db2-d48c-47b3-bf5c-2ff6d65cc65b" containerName="maas-api" containerID="cri-o://2fdb279f03ce351211ac8ea752d561092fe7ce7c8346a3f851f78461d53fa99d" gracePeriod=30 Nov 02 09:58:21.571642 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:21.571609 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:21.894041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:21.894012 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:58:22.009665 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.009627 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-kube-api-access-tzgh7\" (UniqueName: \"kubernetes.io/projected/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-kube-api-access-tzgh7\") pod \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " Nov 02 09:58:22.010156 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.009728 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-maas-api-tls\") pod \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " Nov 02 09:58:22.010156 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.009926 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-openshift-service-ca\") pod \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " Nov 02 09:58:22.010156 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.010021 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-metrics-certs\") pod \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\" (UID: \"62503db2-d48c-47b3-bf5c-2ff6d65cc65b\") " Nov 02 09:58:22.010761 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.010718 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-openshift-service-ca" pod "62503db2-d48c-47b3-bf5c-2ff6d65cc65b" (UID: "62503db2-d48c-47b3-bf5c-2ff6d65cc65b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:22.013298 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.013188 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-kube-api-access-tzgh7" pod "62503db2-d48c-47b3-bf5c-2ff6d65cc65b" (UID: "62503db2-d48c-47b3-bf5c-2ff6d65cc65b"). InnerVolumeSpecName "kube-api-access-tzgh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:22.013298 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.013261 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-maas-api-tls" pod "62503db2-d48c-47b3-bf5c-2ff6d65cc65b" (UID: "62503db2-d48c-47b3-bf5c-2ff6d65cc65b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:58:22.013298 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.013284 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-metrics-certs" pod "62503db2-d48c-47b3-bf5c-2ff6d65cc65b" (UID: "62503db2-d48c-47b3-bf5c-2ff6d65cc65b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:58:22.120967 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.115168 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:22.120967 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.115208 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:22.120967 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.115224 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:22.120967 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.115239 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgh7\" (UniqueName: \"kubernetes.io/projected/62503db2-d48c-47b3-bf5c-2ff6d65cc65b-kube-api-access-tzgh7\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:22.148924 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.146837 4467 scope.go:122] "RemoveContainer" containerID="2fdb279f03ce351211ac8ea752d561092fe7ce7c8346a3f851f78461d53fa99d" Nov 02 09:58:22.148924 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.147081 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:22.233683 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:22.233641 4467 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62503db2_d48c_47b3_bf5c_2ff6d65cc65b.slice/crio-4d3ee647c20cb5403f6e02891ef374ef8f34945f68f2fcfe89d8ea5c7bbab589\": RecentStats: unable to find data in memory cache]" Nov 02 09:58:22.394572 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.394473 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:22.517527 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.514513 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" Nov 02 09:58:22.517527 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.515823 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" event={"ID":"62503db2-d48c-47b3-bf5c-2ff6d65cc65b","Type":"ContainerDied","Data":"2fdb279f03ce351211ac8ea752d561092fe7ce7c8346a3f851f78461d53fa99d"} Nov 02 09:58:22.517527 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.515884 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" event={"ID":"62503db2-d48c-47b3-bf5c-2ff6d65cc65b","Type":"ContainerDied","Data":"4d3ee647c20cb5403f6e02891ef374ef8f34945f68f2fcfe89d8ea5c7bbab589"} Nov 02 09:58:22.565243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.565209 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq"] Nov 02 09:58:22.577453 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:22.577420 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq"] Nov 02 09:58:23.131918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:23.131459 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:23.347326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:23.347278 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62503db2-d48c-47b3-bf5c-2ff6d65cc65b" path="/var/lib/kubelet/pods/62503db2-d48c-47b3-bf5c-2ff6d65cc65b/volumes" Nov 02 09:58:23.877128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:23.874792 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:24.074767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:24.074737 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:24.697248 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:24.697204 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:24.859688 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:24.859648 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2e80fb-57c56d6758-sfmfq" podUID="62503db2-d48c-47b3-bf5c-2ff6d65cc65b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.44:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 09:58:25.540098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:25.540063 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:25.954307 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:25.954264 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" podUID="9417297f-ed2e-4748-8945-2a391a9809bb" containerName="payload-processing" containerID="cri-o://a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8" gracePeriod=25 Nov 02 09:58:25.961034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:25.960965 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" podUID="ba0977d0-2ce6-4177-b59d-966560321782" containerName="payload-processing" containerID="cri-o://f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75" gracePeriod=25 Nov 02 09:58:26.265042 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.264208 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:26.343234 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.343209 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:26.346109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.346088 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:58:26.416812 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.416774 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-kube-api-access-zgrc2\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-kube-api-access-zgrc2\") pod \"9417297f-ed2e-4748-8945-2a391a9809bb\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " Nov 02 09:58:26.417036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.416871 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-plugins-config-volume\") pod \"9417297f-ed2e-4748-8945-2a391a9809bb\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " Nov 02 09:58:26.417036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.416965 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-kube-api-access-wt78b\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-kube-api-access-wt78b\") pod \"ba0977d0-2ce6-4177-b59d-966560321782\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " Nov 02 09:58:26.417036 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.417024 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-provider-credentials\") pod \"ba0977d0-2ce6-4177-b59d-966560321782\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " Nov 02 09:58:26.417225 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.417060 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-routing-overlay-volume\") pod \"9417297f-ed2e-4748-8945-2a391a9809bb\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " Nov 02 09:58:26.417225 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.417099 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-plugins-config-volume\") pod \"ba0977d0-2ce6-4177-b59d-966560321782\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " Nov 02 09:58:26.417344 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.417244 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-provider-credentials\") pod \"9417297f-ed2e-4748-8945-2a391a9809bb\" (UID: \"9417297f-ed2e-4748-8945-2a391a9809bb\") " Nov 02 09:58:26.417465 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.417342 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-routing-overlay-volume\") pod \"ba0977d0-2ce6-4177-b59d-966560321782\" (UID: \"ba0977d0-2ce6-4177-b59d-966560321782\") " Nov 02 09:58:26.418160 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.418108 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-routing-overlay-volume" pod "9417297f-ed2e-4748-8945-2a391a9809bb" (UID: "9417297f-ed2e-4748-8945-2a391a9809bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:26.418292 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.418191 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-routing-overlay-volume" pod "ba0977d0-2ce6-4177-b59d-966560321782" (UID: "ba0977d0-2ce6-4177-b59d-966560321782"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:26.418292 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.418214 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-plugins-config-volume" pod "9417297f-ed2e-4748-8945-2a391a9809bb" (UID: "9417297f-ed2e-4748-8945-2a391a9809bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:26.418423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.418381 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-plugins-config-volume" pod "ba0977d0-2ce6-4177-b59d-966560321782" (UID: "ba0977d0-2ce6-4177-b59d-966560321782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:26.420745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.420693 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-kube-api-access-zgrc2" pod "9417297f-ed2e-4748-8945-2a391a9809bb" (UID: "9417297f-ed2e-4748-8945-2a391a9809bb"). InnerVolumeSpecName "kube-api-access-zgrc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:26.420865 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.420765 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-kube-api-access-wt78b" pod "ba0977d0-2ce6-4177-b59d-966560321782" (UID: "ba0977d0-2ce6-4177-b59d-966560321782"). InnerVolumeSpecName "kube-api-access-wt78b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:26.421159 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.421128 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-provider-credentials" pod "ba0977d0-2ce6-4177-b59d-966560321782" (UID: "ba0977d0-2ce6-4177-b59d-966560321782"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:26.422344 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.422316 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-provider-credentials" pod "9417297f-ed2e-4748-8945-2a391a9809bb" (UID: "9417297f-ed2e-4748-8945-2a391a9809bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:26.519835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.519746 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:26.519835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.519788 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgrc2\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-kube-api-access-zgrc2\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:26.519835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.519805 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:26.519835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.519821 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt78b\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-kube-api-access-wt78b\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:26.519835 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.519835 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba0977d0-2ce6-4177-b59d-966560321782-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:26.520139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.519851 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9417297f-ed2e-4748-8945-2a391a9809bb-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:26.520139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.519866 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0977d0-2ce6-4177-b59d-966560321782-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:26.520139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.519880 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9417297f-ed2e-4748-8945-2a391a9809bb-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:26.551147 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.551116 4467 generic.go:356] "Generic (PLEG): container finished" podID="9417297f-ed2e-4748-8945-2a391a9809bb" containerID="a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8" exitCode=0 Nov 02 09:58:26.551338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.551160 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" Nov 02 09:58:26.551338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.551206 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" event={"ID":"9417297f-ed2e-4748-8945-2a391a9809bb","Type":"ContainerDied","Data":"a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8"} Nov 02 09:58:26.551338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.551267 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk" event={"ID":"9417297f-ed2e-4748-8945-2a391a9809bb","Type":"ContainerDied","Data":"c6cfaba82bcef3c14f995909a9ef6c26a328e4dfe9cf4efc46ffa07bc314e952"} Nov 02 09:58:26.551338 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.551298 4467 scope.go:122] "RemoveContainer" containerID="a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8" Nov 02 09:58:26.552732 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.552713 4467 generic.go:356] "Generic (PLEG): container finished" podID="ba0977d0-2ce6-4177-b59d-966560321782" containerID="f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75" exitCode=0 Nov 02 09:58:26.552838 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.552759 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" Nov 02 09:58:26.552838 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.552786 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" event={"ID":"ba0977d0-2ce6-4177-b59d-966560321782","Type":"ContainerDied","Data":"f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75"} Nov 02 09:58:26.552838 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.552836 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv" event={"ID":"ba0977d0-2ce6-4177-b59d-966560321782","Type":"ContainerDied","Data":"ec5449249693c58188c638206987bbbddeb3de22787fe1a861d1f138956c9661"} Nov 02 09:58:26.562266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.562250 4467 scope.go:122] "RemoveContainer" containerID="a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8" Nov 02 09:58:26.562532 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:26.562515 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8\": container with ID starting with a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8 not found: ID does not exist" containerID="a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8" Nov 02 09:58:26.562620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.562537 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8"} err="failed to get container status \"a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8\": rpc error: code = NotFound desc = could not find container \"a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8\": container with ID starting with a5049ea87e1e4ca333c1a4b1d0479b17ab98c5ecda3465fa7c3e230c1448bac8 not found: ID does not exist" Nov 02 09:58:26.562620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.562556 4467 scope.go:122] "RemoveContainer" containerID="f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75" Nov 02 09:58:26.573435 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.572014 4467 scope.go:122] "RemoveContainer" containerID="f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75" Nov 02 09:58:26.574942 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:26.574904 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75\": container with ID starting with f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75 not found: ID does not exist" containerID="f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75" Nov 02 09:58:26.575074 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.574947 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75"} err="failed to get container status \"f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75\": rpc error: code = NotFound desc = could not find container \"f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75\": container with ID starting with f96c08f3f20900f21d93f372b3823da827504bec9b73b313727e2f502689ec75 not found: ID does not exist" Nov 02 09:58:26.578195 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.578172 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk"] Nov 02 09:58:26.581493 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.581472 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-ndszk"] Nov 02 09:58:26.593106 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.593084 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv"] Nov 02 09:58:26.597681 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.597661 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff7768868-lfrbv"] Nov 02 09:58:26.857991 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:26.857959 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:58:26.952547 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:26.952512 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:27.333831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:27.333796 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9417297f-ed2e-4748-8945-2a391a9809bb" path="/var/lib/kubelet/pods/9417297f-ed2e-4748-8945-2a391a9809bb/volumes" Nov 02 09:58:27.334342 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:27.334325 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0977d0-2ce6-4177-b59d-966560321782" path="/var/lib/kubelet/pods/ba0977d0-2ce6-4177-b59d-966560321782/volumes" Nov 02 09:58:27.683985 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:27.682434 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:29.031915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:29.028820 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:58:29.762373 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:29.762341 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:29.970309 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:29.970275 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:31.204913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:31.204732 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:32.356442 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:32.355221 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:33.273135 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.273018 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb"] Nov 02 09:58:33.273359 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.273272 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" podUID="90ee45a7-858e-4e54-93b9-acb0f12be41c" containerName="maas-api" containerID="cri-o://2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e" gracePeriod=30 Nov 02 09:58:33.404000 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.403753 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:33.632620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.626562 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:33.633920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.633300 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.724993 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg"] Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.725943 4467 cpu_manager.go:414] "containerMap: removing container" podUID="ba0977d0-2ce6-4177-b59d-966560321782" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.725968 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0977d0-2ce6-4177-b59d-966560321782" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.725991 4467 cpu_manager.go:414] "containerMap: removing container" podUID="9417297f-ed2e-4748-8945-2a391a9809bb" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726000 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9417297f-ed2e-4748-8945-2a391a9809bb" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726015 4467 cpu_manager.go:414] "containerMap: removing container" podUID="90ee45a7-858e-4e54-93b9-acb0f12be41c" containerName="maas-api" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726023 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ee45a7-858e-4e54-93b9-acb0f12be41c" containerName="maas-api" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726037 4467 cpu_manager.go:414] "containerMap: removing container" podUID="22f1ed85-8366-42bd-87a1-97fc1cf92825" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726046 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f1ed85-8366-42bd-87a1-97fc1cf92825" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726076 4467 cpu_manager.go:414] "containerMap: removing container" podUID="62503db2-d48c-47b3-bf5c-2ff6d65cc65b" containerName="maas-api" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726084 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62503db2-d48c-47b3-bf5c-2ff6d65cc65b" containerName="maas-api" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726238 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0977d0-2ce6-4177-b59d-966560321782" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726253 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="9417297f-ed2e-4748-8945-2a391a9809bb" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726262 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ee45a7-858e-4e54-93b9-acb0f12be41c" containerName="maas-api" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726272 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f1ed85-8366-42bd-87a1-97fc1cf92825" containerName="payload-processing" Nov 02 09:58:33.727528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.726283 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="62503db2-d48c-47b3-bf5c-2ff6d65cc65b" containerName="maas-api" Nov 02 09:58:33.736091 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.735971 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.748736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.745390 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-maas-api-tls\") pod \"90ee45a7-858e-4e54-93b9-acb0f12be41c\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " Nov 02 09:58:33.748736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.745655 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ee45a7-858e-4e54-93b9-acb0f12be41c-kube-api-access-cv7vb\" (UniqueName: \"kubernetes.io/projected/90ee45a7-858e-4e54-93b9-acb0f12be41c-kube-api-access-cv7vb\") pod \"90ee45a7-858e-4e54-93b9-acb0f12be41c\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " Nov 02 09:58:33.748736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.745752 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-metrics-certs\") pod \"90ee45a7-858e-4e54-93b9-acb0f12be41c\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " Nov 02 09:58:33.748736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.745785 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ee45a7-858e-4e54-93b9-acb0f12be41c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ee45a7-858e-4e54-93b9-acb0f12be41c-openshift-service-ca\") pod \"90ee45a7-858e-4e54-93b9-acb0f12be41c\" (UID: \"90ee45a7-858e-4e54-93b9-acb0f12be41c\") " Nov 02 09:58:33.748736 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.748402 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ee45a7-858e-4e54-93b9-acb0f12be41c-openshift-service-ca" pod "90ee45a7-858e-4e54-93b9-acb0f12be41c" (UID: "90ee45a7-858e-4e54-93b9-acb0f12be41c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:33.755102 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.754643 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ee45a7-858e-4e54-93b9-acb0f12be41c-kube-api-access-cv7vb" pod "90ee45a7-858e-4e54-93b9-acb0f12be41c" (UID: "90ee45a7-858e-4e54-93b9-acb0f12be41c"). InnerVolumeSpecName "kube-api-access-cv7vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:33.760323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.760274 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-metrics-certs" pod "90ee45a7-858e-4e54-93b9-acb0f12be41c" (UID: "90ee45a7-858e-4e54-93b9-acb0f12be41c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:58:33.765580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.763972 4467 generic.go:356] "Generic (PLEG): container finished" podID="90ee45a7-858e-4e54-93b9-acb0f12be41c" containerID="2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e" exitCode=0 Nov 02 09:58:33.765580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.764093 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" event={"ID":"90ee45a7-858e-4e54-93b9-acb0f12be41c","Type":"ContainerDied","Data":"2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e"} Nov 02 09:58:33.765580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.764142 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" event={"ID":"90ee45a7-858e-4e54-93b9-acb0f12be41c","Type":"ContainerDied","Data":"331bfbd3f16d35fe256c606fdbf3b4323af967d4a05d33cedacd3cfd2e48ae07"} Nov 02 09:58:33.765580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.764182 4467 scope.go:122] "RemoveContainer" containerID="2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e" Nov 02 09:58:33.765580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.764351 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb" Nov 02 09:58:33.824599 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.824569 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg"] Nov 02 09:58:33.826078 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.826024 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-maas-api-tls" pod "90ee45a7-858e-4e54-93b9-acb0f12be41c" (UID: "90ee45a7-858e-4e54-93b9-acb0f12be41c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:58:33.855229 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.855202 4467 scope.go:122] "RemoveContainer" containerID="2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e" Nov 02 09:58:33.865185 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.864948 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.865185 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.865090 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.865185 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.865155 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn5dp\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-kube-api-access-pn5dp\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.865474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.865424 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.865561 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.865538 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv7vb\" (UniqueName: \"kubernetes.io/projected/90ee45a7-858e-4e54-93b9-acb0f12be41c-kube-api-access-cv7vb\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:33.866551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.865569 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:33.866551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.865585 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ee45a7-858e-4e54-93b9-acb0f12be41c-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:33.866551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.865599 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ee45a7-858e-4e54-93b9-acb0f12be41c-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:33.871139 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:33.867057 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e\": container with ID starting with 2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e not found: ID does not exist" containerID="2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e" Nov 02 09:58:33.871139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.867116 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e"} err="failed to get container status \"2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e\": rpc error: code = NotFound desc = could not find container \"2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e\": container with ID starting with 2d3f71cf255ffba3b95cb76fa61b61b4a5f35ee83bdd110440b0041574b9b17e not found: ID does not exist" Nov 02 09:58:33.871139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.867387 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:33.966864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.966827 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.967073 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.966946 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.967073 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.967042 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.967204 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.967082 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn5dp\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-kube-api-access-pn5dp\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.967713 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.967684 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.967813 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.967775 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.969565 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.969543 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:33.979692 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:33.979669 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn5dp\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-kube-api-access-pn5dp\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:34.120266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.119959 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:34.164296 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.164264 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb"] Nov 02 09:58:34.172411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.172347 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-bde944-cf8fb9598-gllcb"] Nov 02 09:58:34.480089 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.479984 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg"] Nov 02 09:58:34.499135 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.498987 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:34.499406 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:58:34.499372 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode491d8b2_2991_4fab_a419_29ca27fb58b2.slice/crio-efcdf33a13a73a178c482496c517a9965adb047f5d61150e4fcf4efeaaedbc72 WatchSource:0}: Error finding container efcdf33a13a73a178c482496c517a9965adb047f5d61150e4fcf4efeaaedbc72: Status 404 returned error can't find the container with id efcdf33a13a73a178c482496c517a9965adb047f5d61150e4fcf4efeaaedbc72 Nov 02 09:58:34.771846 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.771748 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" event={"ID":"e491d8b2-2991-4fab-a419-29ca27fb58b2","Type":"ContainerStarted","Data":"10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42"} Nov 02 09:58:34.771846 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.771811 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:34.771846 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.771835 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" event={"ID":"e491d8b2-2991-4fab-a419-29ca27fb58b2","Type":"ContainerStarted","Data":"efcdf33a13a73a178c482496c517a9965adb047f5d61150e4fcf4efeaaedbc72"} Nov 02 09:58:34.794151 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:34.794086 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" podStartSLOduration=1.794072697 podStartE2EDuration="1.794072697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:34.791665847 +0000 UTC m=+1393.631607845" watchObservedRunningTime="2026-11-02 09:58:34.794072697 +0000 UTC m=+1393.634014692" Nov 02 09:58:35.334282 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:35.334248 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ee45a7-858e-4e54-93b9-acb0f12be41c" path="/var/lib/kubelet/pods/90ee45a7-858e-4e54-93b9-acb0f12be41c/volumes" Nov 02 09:58:35.440468 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:35.440422 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:35.756795 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:35.756712 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:35.939957 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:35.939921 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:36.410677 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.410643 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:36.599236 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.599206 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:36.669405 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.669317 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds"] Nov 02 09:58:36.669597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.669571 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" containerID="cri-o://ad925f30939d1b739816cbd573ffa2b2da6a2a87211f6db6afba83add7c0000b" gracePeriod=30 Nov 02 09:58:36.688904 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.688864 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf"] Nov 02 09:58:36.689364 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.689297 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" containerID="cri-o://1302bfd5f6eed8d071513c33c8fb1371210e3b4b611f1ba69b7fd02576be57a2" gracePeriod=30 Nov 02 09:58:36.692842 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.692821 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2"] Nov 02 09:58:36.693067 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.693046 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" containerID="cri-o://d90621427d7a6d8725808b02edcf5504cd273a087bcddcc032f7f64cd1d7539b" gracePeriod=30 Nov 02 09:58:36.829177 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:36.829141 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:37.456831 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:37.456797 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:37.654379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:37.654343 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:38.154620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:38.154585 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:38.854438 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:38.854408 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:39.246403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:39.246309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:39.436346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:39.436316 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:39.698591 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:39.698556 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:39.911923 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:39.906790 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:40.111848 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:40.111802 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:40.955198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:40.955140 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:41.591039 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.590989 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:41.811659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.811631 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:41.866259 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.866230 4467 generic.go:356] "Generic (PLEG): container finished" podID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerID="1302bfd5f6eed8d071513c33c8fb1371210e3b4b611f1ba69b7fd02576be57a2" exitCode=0 Nov 02 09:58:41.866423 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.866402 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" event={"ID":"7aed7e46-1726-4cbc-bac5-45a69cbce271","Type":"ContainerDied","Data":"1302bfd5f6eed8d071513c33c8fb1371210e3b4b611f1ba69b7fd02576be57a2"} Nov 02 09:58:41.868072 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.868045 4467 generic.go:356] "Generic (PLEG): container finished" podID="1a918367-c36f-4816-8674-c33e3beb85ce" containerID="d90621427d7a6d8725808b02edcf5504cd273a087bcddcc032f7f64cd1d7539b" exitCode=0 Nov 02 09:58:41.868200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.868172 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" event={"ID":"1a918367-c36f-4816-8674-c33e3beb85ce","Type":"ContainerDied","Data":"d90621427d7a6d8725808b02edcf5504cd273a087bcddcc032f7f64cd1d7539b"} Nov 02 09:58:41.869715 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.869696 4467 generic.go:356] "Generic (PLEG): container finished" podID="57315ea7-23ff-442b-8302-c909afc112cc" containerID="ad925f30939d1b739816cbd573ffa2b2da6a2a87211f6db6afba83add7c0000b" exitCode=0 Nov 02 09:58:41.869824 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.869732 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" event={"ID":"57315ea7-23ff-442b-8302-c909afc112cc","Type":"ContainerDied","Data":"ad925f30939d1b739816cbd573ffa2b2da6a2a87211f6db6afba83add7c0000b"} Nov 02 09:58:41.995954 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:41.995746 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:42.029384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.029016 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:58:42.091377 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.091139 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:58:42.121522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.121454 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.152495 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-kube-api-access-xmtm7\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-kube-api-access-xmtm7\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.155466 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57315ea7-23ff-442b-8302-c909afc112cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57315ea7-23ff-442b-8302-c909afc112cc-istio-podinfo\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.155534 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-data\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.155621 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-envoy\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.155672 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-credential-socket\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.155724 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-socket\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.155800 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57315ea7-23ff-442b-8302-c909afc112cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57315ea7-23ff-442b-8302-c909afc112cc-istiod-ca-cert\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.155859 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-istio-token\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-istio-token\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.155936 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-certs\") pod \"57315ea7-23ff-442b-8302-c909afc112cc\" (UID: \"57315ea7-23ff-442b-8302-c909afc112cc\") " Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.156787 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-certs" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.157269 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-credential-socket" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.157439 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-data" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.157911 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-socket" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.161639 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.158197 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57315ea7-23ff-442b-8302-c909afc112cc-istiod-ca-cert" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:42.170419 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.162317 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-envoy" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.170419 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.162426 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57315ea7-23ff-442b-8302-c909afc112cc-istio-podinfo" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:42.170419 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.167882 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-istio-token" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:42.171296 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.170990 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc"] Nov 02 09:58:42.177656 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.173318 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" containerID="cri-o://d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d" gracePeriod=30 Nov 02 09:58:42.197238 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.197193 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-kube-api-access-xmtm7" pod "57315ea7-23ff-442b-8302-c909afc112cc" (UID: "57315ea7-23ff-442b-8302-c909afc112cc"). InnerVolumeSpecName "kube-api-access-xmtm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:42.197814 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.197795 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8"] Nov 02 09:58:42.198306 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.198276 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" containerID="cri-o://9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851" gracePeriod=30 Nov 02 09:58:42.206358 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.206314 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf"] Nov 02 09:58:42.207124 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.206939 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" containerID="cri-o://21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877" gracePeriod=30 Nov 02 09:58:42.261708 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.261670 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-istio-token\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-istio-token\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.292061 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.292019 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-credential-socket\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.293273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.292399 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-envoy\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.304420 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.304385 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a918367-c36f-4816-8674-c33e3beb85ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918367-c36f-4816-8674-c33e3beb85ce-istio-podinfo\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.312023 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.311976 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-certs\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.312293 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.312276 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-token\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-token\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.321424 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.313461 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.323469 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-kube-api-access-sch6q\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-kube-api-access-sch6q\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.323542 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-certs\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.323687 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-envoy\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.323729 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a918367-c36f-4816-8674-c33e3beb85ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918367-c36f-4816-8674-c33e3beb85ce-istiod-ca-cert\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.323824 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-data\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.323873 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-credential-socket\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.323945 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-socket\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.323987 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aed7e46-1726-4cbc-bac5-45a69cbce271-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aed7e46-1726-4cbc-bac5-45a69cbce271-istiod-ca-cert\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324016 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-socket\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324050 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-podinfo\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324082 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-data\") pod \"7aed7e46-1726-4cbc-bac5-45a69cbce271\" (UID: \"7aed7e46-1726-4cbc-bac5-45a69cbce271\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324123 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-kube-api-access-c867x\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-kube-api-access-c867x\") pod \"1a918367-c36f-4816-8674-c33e3beb85ce\" (UID: \"1a918367-c36f-4816-8674-c33e3beb85ce\") " Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324733 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324753 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57315ea7-23ff-442b-8302-c909afc112cc-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324766 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324778 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324791 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmtm7\" (UniqueName: \"kubernetes.io/projected/57315ea7-23ff-442b-8302-c909afc112cc-kube-api-access-xmtm7\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324805 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57315ea7-23ff-442b-8302-c909afc112cc-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324816 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324831 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.325165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.324844 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57315ea7-23ff-442b-8302-c909afc112cc-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.330979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.294113 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-credential-socket" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.330979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.300002 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-envoy" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.330979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.313764 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-istio-token" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:42.330979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.314203 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-certs" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.330979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.327085 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-token" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:42.330979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.327312 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-credential-socket" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.330979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.330687 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a918367-c36f-4816-8674-c33e3beb85ce-istio-podinfo" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:42.331394 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.331151 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-socket" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.331394 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.331332 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-socket" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.331714 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.331687 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aed7e46-1726-4cbc-bac5-45a69cbce271-istiod-ca-cert" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:42.335323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.335296 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a918367-c36f-4816-8674-c33e3beb85ce-istiod-ca-cert" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:42.339080 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.339049 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-certs" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.347479 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.347439 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-data" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.347709 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.347684 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-data" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.359764 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.359736 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-podinfo" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:42.375462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.374771 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-envoy" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:42.375462 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.375354 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-kube-api-access-sch6q" pod "7aed7e46-1726-4cbc-bac5-45a69cbce271" (UID: "7aed7e46-1726-4cbc-bac5-45a69cbce271"). InnerVolumeSpecName "kube-api-access-sch6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:42.378110 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.378074 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-kube-api-access-c867x" pod "1a918367-c36f-4816-8674-c33e3beb85ce" (UID: "1a918367-c36f-4816-8674-c33e3beb85ce"). InnerVolumeSpecName "kube-api-access-c867x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427258 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427293 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427308 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427321 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aed7e46-1726-4cbc-bac5-45a69cbce271-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427335 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427348 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427363 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427378 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c867x\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-kube-api-access-c867x\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427394 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918367-c36f-4816-8674-c33e3beb85ce-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427407 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427422 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427435 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918367-c36f-4816-8674-c33e3beb85ce-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427450 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918367-c36f-4816-8674-c33e3beb85ce-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427464 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427484 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sch6q\" (UniqueName: \"kubernetes.io/projected/7aed7e46-1726-4cbc-bac5-45a69cbce271-kube-api-access-sch6q\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427499 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427515 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aed7e46-1726-4cbc-bac5-45a69cbce271-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.427558 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.427530 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918367-c36f-4816-8674-c33e3beb85ce-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:42.690247 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.690164 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:42.880488 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.880447 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:42.892942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.892084 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" Nov 02 09:58:42.892942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.892107 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf" event={"ID":"7aed7e46-1726-4cbc-bac5-45a69cbce271","Type":"ContainerDied","Data":"71c407c8a8cca95430c67d70940e8d86530925275d7d9bcf83f4e689c7452b65"} Nov 02 09:58:42.892942 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.892184 4467 scope.go:122] "RemoveContainer" containerID="1302bfd5f6eed8d071513c33c8fb1371210e3b4b611f1ba69b7fd02576be57a2" Nov 02 09:58:42.894554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.894527 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" event={"ID":"1a918367-c36f-4816-8674-c33e3beb85ce","Type":"ContainerDied","Data":"4de6eeb2122f27ffa89598decfa2f7b5e2fdbb96815f5a4cc219b9d36cd970ea"} Nov 02 09:58:42.894660 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.894538 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2" Nov 02 09:58:42.899952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.899924 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" event={"ID":"57315ea7-23ff-442b-8302-c909afc112cc","Type":"ContainerDied","Data":"cf293394f0b8be81a75c107954e22b7e9386ac7feb9b04215a845ad97dda5517"} Nov 02 09:58:42.900081 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.900034 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds" Nov 02 09:58:42.909188 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.909071 4467 scope.go:122] "RemoveContainer" containerID="d90621427d7a6d8725808b02edcf5504cd273a087bcddcc032f7f64cd1d7539b" Nov 02 09:58:42.923070 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.923045 4467 scope.go:122] "RemoveContainer" containerID="ad925f30939d1b739816cbd573ffa2b2da6a2a87211f6db6afba83add7c0000b" Nov 02 09:58:42.923163 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.923075 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf"] Nov 02 09:58:42.928259 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.928238 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zt2tf"] Nov 02 09:58:42.941920 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.941819 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds"] Nov 02 09:58:42.953098 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.953068 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-vq4ds"] Nov 02 09:58:42.967773 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.967740 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2"] Nov 02 09:58:42.972200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:42.971819 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-58jd2"] Nov 02 09:58:43.238999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:43.238854 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:43.332311 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:43.332279 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" path="/var/lib/kubelet/pods/1a918367-c36f-4816-8674-c33e3beb85ce/volumes" Nov 02 09:58:43.332781 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:43.332766 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57315ea7-23ff-442b-8302-c909afc112cc" path="/var/lib/kubelet/pods/57315ea7-23ff-442b-8302-c909afc112cc/volumes" Nov 02 09:58:43.333196 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:43.333184 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" path="/var/lib/kubelet/pods/7aed7e46-1726-4cbc-bac5-45a69cbce271/volumes" Nov 02 09:58:44.070144 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:44.070109 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:44.769728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:44.769700 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:45.381345 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:45.381313 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:58:45.571112 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:45.571080 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:45.782280 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:45.782203 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:58:45.817925 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:45.817870 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:45.838278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:45.838245 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg"] Nov 02 09:58:45.949482 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:45.949445 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 09:58:45.949708 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:45.949678 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 09:58:46.150165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.150121 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l"] Nov 02 09:58:46.154440 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.154376 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" podUID="7f08e396-bdfc-46c6-9403-1134d5750473" containerName="maas-api" containerID="cri-o://5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec" gracePeriod=30 Nov 02 09:58:46.275201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.275168 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:46.550739 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.550708 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:58:46.622244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.622208 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:46.625534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.625498 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-maas-api-tls\") pod \"7f08e396-bdfc-46c6-9403-1134d5750473\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " Nov 02 09:58:46.625678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.625586 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f08e396-bdfc-46c6-9403-1134d5750473-kube-api-access-bv25g\" (UniqueName: \"kubernetes.io/projected/7f08e396-bdfc-46c6-9403-1134d5750473-kube-api-access-bv25g\") pod \"7f08e396-bdfc-46c6-9403-1134d5750473\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " Nov 02 09:58:46.625678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.625629 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-metrics-certs\") pod \"7f08e396-bdfc-46c6-9403-1134d5750473\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " Nov 02 09:58:46.625678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.625660 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f08e396-bdfc-46c6-9403-1134d5750473-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f08e396-bdfc-46c6-9403-1134d5750473-openshift-service-ca\") pod \"7f08e396-bdfc-46c6-9403-1134d5750473\" (UID: \"7f08e396-bdfc-46c6-9403-1134d5750473\") " Nov 02 09:58:46.626412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.626379 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f08e396-bdfc-46c6-9403-1134d5750473-openshift-service-ca" pod "7f08e396-bdfc-46c6-9403-1134d5750473" (UID: "7f08e396-bdfc-46c6-9403-1134d5750473"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:46.628204 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.628178 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f08e396-bdfc-46c6-9403-1134d5750473-kube-api-access-bv25g" pod "7f08e396-bdfc-46c6-9403-1134d5750473" (UID: "7f08e396-bdfc-46c6-9403-1134d5750473"). InnerVolumeSpecName "kube-api-access-bv25g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:46.628597 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.628583 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-maas-api-tls" pod "7f08e396-bdfc-46c6-9403-1134d5750473" (UID: "7f08e396-bdfc-46c6-9403-1134d5750473"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:58:46.629217 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.629195 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-metrics-certs" pod "7f08e396-bdfc-46c6-9403-1134d5750473" (UID: "7f08e396-bdfc-46c6-9403-1134d5750473"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 09:58:46.727015 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.726945 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv25g\" (UniqueName: \"kubernetes.io/projected/7f08e396-bdfc-46c6-9403-1134d5750473-kube-api-access-bv25g\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:46.727015 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.727005 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:46.727015 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.727017 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f08e396-bdfc-46c6-9403-1134d5750473-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:46.727015 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.727026 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f08e396-bdfc-46c6-9403-1134d5750473-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:46.822457 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.822426 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 09:58:46.822981 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.822489 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 09:58:46.928659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.928616 4467 generic.go:356] "Generic (PLEG): container finished" podID="7f08e396-bdfc-46c6-9403-1134d5750473" containerID="5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec" exitCode=0 Nov 02 09:58:46.928659 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.928657 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" event={"ID":"7f08e396-bdfc-46c6-9403-1134d5750473","Type":"ContainerDied","Data":"5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec"} Nov 02 09:58:46.928884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.928664 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" Nov 02 09:58:46.928884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.928701 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" event={"ID":"7f08e396-bdfc-46c6-9403-1134d5750473","Type":"ContainerDied","Data":"6a56112dddcb1c400cb1198b198bc810a81a7d03fd774a9ed1006234059dd5dd"} Nov 02 09:58:46.928884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.928725 4467 scope.go:122] "RemoveContainer" containerID="5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec" Nov 02 09:58:46.939540 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.939518 4467 scope.go:122] "RemoveContainer" containerID="5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec" Nov 02 09:58:46.939851 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:46.939831 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec\": container with ID starting with 5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec not found: ID does not exist" containerID="5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec" Nov 02 09:58:46.939952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.939858 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec"} err="failed to get container status \"5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec\": rpc error: code = NotFound desc = could not find container \"5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec\": container with ID starting with 5bacd04c443cfa01e43d72bdb0018bc7a9e17bd1ffc54bb82458f16b7fc5efec not found: ID does not exist" Nov 02 09:58:46.955083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.955054 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l"] Nov 02 09:58:46.959215 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:46.959188 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l"] Nov 02 09:58:47.243449 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.243415 4467 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 09:58:47.243589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.243480 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 09:58:47.343467 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.343436 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f08e396-bdfc-46c6-9403-1134d5750473" path="/var/lib/kubelet/pods/7f08e396-bdfc-46c6-9403-1134d5750473/volumes" Nov 02 09:58:47.399221 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.399140 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:47.556611 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.556576 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:58:47.626762 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.626673 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642136 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istiod-ca-cert\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642193 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-certs\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642236 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-kube-api-access-8fvm2\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-kube-api-access-8fvm2\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642264 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-socket\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642304 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-podinfo\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642386 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-token\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-token\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642422 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-envoy\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642453 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-credential-socket\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642501 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-kube-api-access-zpvsn\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-kube-api-access-zpvsn\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642539 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-credential-socket\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642569 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-token\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-token\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642602 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-podinfo\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642633 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-socket\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642669 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-data\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642698 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-envoy\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642727 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-data\") pod \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\" (UID: \"b78bbcb8-1829-4f6a-8738-9c69969fb66b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642754 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-certs\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.642800 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istiod-ca-cert\") pod \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\" (UID: \"4e7de95e-9a9d-4f12-b7a2-b0f84902240b\") " Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.643029 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-credential-socket" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.643337 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.643363 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istiod-ca-cert" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:47.644953 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.643632 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-certs" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.646137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.645361 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-credential-socket" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.646137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.645411 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-socket" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.646137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.645585 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-data" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.646137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.645625 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-socket" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.646137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.645658 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-data" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.646137 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.645839 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-certs" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.646325 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.646156 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istiod-ca-cert" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:47.648089 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.648022 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-token" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:47.655347 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.652426 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:58:47.658753 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.658720 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-envoy" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.659275 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.659248 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-envoy" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.659593 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.659471 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-podinfo" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:47.659676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.659647 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-token" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:47.660355 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.660333 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-kube-api-access-8fvm2" pod "b78bbcb8-1829-4f6a-8738-9c69969fb66b" (UID: "b78bbcb8-1829-4f6a-8738-9c69969fb66b"). InnerVolumeSpecName "kube-api-access-8fvm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:47.661117 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.661083 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-kube-api-access-zpvsn" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "kube-api-access-zpvsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:47.674668 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.666238 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-podinfo" pod "4e7de95e-9a9d-4f12-b7a2-b0f84902240b" (UID: "4e7de95e-9a9d-4f12-b7a2-b0f84902240b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:47.744176 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744140 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-certs\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.744393 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744209 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-credential-socket\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.744393 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744261 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-kube-api-access-hzkx9\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-kube-api-access-hzkx9\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.744393 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744333 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-podinfo\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.744393 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744378 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1afc788-08f2-4260-af9e-5b3a537133ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1afc788-08f2-4260-af9e-5b3a537133ef-istiod-ca-cert\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.744608 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744416 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-socket\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.744608 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744504 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-data\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.744608 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744533 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-token\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-token\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.744608 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744569 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-envoy\") pod \"b1afc788-08f2-4260-af9e-5b3a537133ef\" (UID: \"b1afc788-08f2-4260-af9e-5b3a537133ef\") " Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.744994 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745014 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745026 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fvm2\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-kube-api-access-8fvm2\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745040 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745051 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745062 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745073 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745084 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpvsn\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-kube-api-access-zpvsn\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745095 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745105 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745116 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745127 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745137 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745150 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745163 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78bbcb8-1829-4f6a-8738-9c69969fb66b-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745175 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745185 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7de95e-9a9d-4f12-b7a2-b0f84902240b-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745219 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-credential-socket" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745445 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-certs" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.745962 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-socket" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.746115 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1afc788-08f2-4260-af9e-5b3a537133ef-istiod-ca-cert" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:47.746316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.746175 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-data" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.748049 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.748021 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-token" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:47.748191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.748153 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-envoy" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:47.748346 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.748320 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-kube-api-access-hzkx9" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "kube-api-access-hzkx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:47.750006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.749968 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-podinfo" pod "b1afc788-08f2-4260-af9e-5b3a537133ef" (UID: "b1afc788-08f2-4260-af9e-5b3a537133ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:47.847063 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847017 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.847063 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847060 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.847481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847076 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzkx9\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-kube-api-access-hzkx9\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.847481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847095 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.847481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847110 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1afc788-08f2-4260-af9e-5b3a537133ef-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.847481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847123 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.847481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847142 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.847481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847156 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.847481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.847170 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1afc788-08f2-4260-af9e-5b3a537133ef-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:47.931970 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.931931 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:47.937337 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.937302 4467 generic.go:356] "Generic (PLEG): container finished" podID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerID="21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877" exitCode=0 Nov 02 09:58:47.937528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.937358 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" Nov 02 09:58:47.937528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.937406 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" event={"ID":"4e7de95e-9a9d-4f12-b7a2-b0f84902240b","Type":"ContainerDied","Data":"21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877"} Nov 02 09:58:47.937528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.937451 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf" event={"ID":"4e7de95e-9a9d-4f12-b7a2-b0f84902240b","Type":"ContainerDied","Data":"b139ab62575b2f86d2b5561f6786b09898f0c8d24faa94b30757891064cd1e50"} Nov 02 09:58:47.937528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.937479 4467 scope.go:122] "RemoveContainer" containerID="21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877" Nov 02 09:58:47.950784 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.950753 4467 generic.go:356] "Generic (PLEG): container finished" podID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerID="9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851" exitCode=0 Nov 02 09:58:47.950999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.950812 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" Nov 02 09:58:47.950999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.950852 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" event={"ID":"b78bbcb8-1829-4f6a-8738-9c69969fb66b","Type":"ContainerDied","Data":"9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851"} Nov 02 09:58:47.950999 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.950939 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8" event={"ID":"b78bbcb8-1829-4f6a-8738-9c69969fb66b","Type":"ContainerDied","Data":"5875dcf2f0d4af5a5c0c580479794a504046a051b851a9544757a22f6fd3c3c8"} Nov 02 09:58:47.955157 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.955131 4467 generic.go:356] "Generic (PLEG): container finished" podID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerID="d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d" exitCode=0 Nov 02 09:58:47.955262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.955176 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" Nov 02 09:58:47.955262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.955184 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" event={"ID":"b1afc788-08f2-4260-af9e-5b3a537133ef","Type":"ContainerDied","Data":"d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d"} Nov 02 09:58:47.955262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.955223 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc" event={"ID":"b1afc788-08f2-4260-af9e-5b3a537133ef","Type":"ContainerDied","Data":"485e0b2d610d395a355f05417883200ead0d997408ec44a2f5e66f1e4fb8cf6d"} Nov 02 09:58:47.957709 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.957690 4467 scope.go:122] "RemoveContainer" containerID="21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877" Nov 02 09:58:47.958012 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:47.957990 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877\": container with ID starting with 21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877 not found: ID does not exist" containerID="21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877" Nov 02 09:58:47.958068 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.958025 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877"} err="failed to get container status \"21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877\": rpc error: code = NotFound desc = could not find container \"21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877\": container with ID starting with 21520f18fa8e7817d9077c720ae0c302f5e84c4a52812d88e1a29600c8ff3877 not found: ID does not exist" Nov 02 09:58:47.958068 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.958048 4467 scope.go:122] "RemoveContainer" containerID="9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851" Nov 02 09:58:47.974286 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.974265 4467 scope.go:122] "RemoveContainer" containerID="9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851" Nov 02 09:58:47.974393 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.974355 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf"] Nov 02 09:58:47.975738 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.975717 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-k6wmf"] Nov 02 09:58:47.978233 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:47.978207 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851\": container with ID starting with 9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851 not found: ID does not exist" containerID="9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851" Nov 02 09:58:47.978333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.978235 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851"} err="failed to get container status \"9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851\": rpc error: code = NotFound desc = could not find container \"9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851\": container with ID starting with 9e420cc3acdfd8fd3d2d7c915b7d764624b2496cbffcb2878db7ae4e42468851 not found: ID does not exist" Nov 02 09:58:47.978333 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.978253 4467 scope.go:122] "RemoveContainer" containerID="d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d" Nov 02 09:58:47.987800 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.987777 4467 scope.go:122] "RemoveContainer" containerID="d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d" Nov 02 09:58:47.988399 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:47.988363 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d\": container with ID starting with d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d not found: ID does not exist" containerID="d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d" Nov 02 09:58:47.988503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.988395 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d"} err="failed to get container status \"d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d\": rpc error: code = NotFound desc = could not find container \"d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d\": container with ID starting with d3bf15a0eda9170e1b1da020a48a7c4dc4954ef4613dd68c688935f1aee2336d not found: ID does not exist" Nov 02 09:58:47.994565 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.994538 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8"] Nov 02 09:58:47.998534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:47.998508 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-ndhs8"] Nov 02 09:58:48.009109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:48.009080 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc"] Nov 02 09:58:48.015883 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:48.015847 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bssvc"] Nov 02 09:58:48.107498 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:48.107408 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:48.964187 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:48.964151 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:49.332833 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:49.332802 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" path="/var/lib/kubelet/pods/4e7de95e-9a9d-4f12-b7a2-b0f84902240b/volumes" Nov 02 09:58:49.333302 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:49.333288 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" path="/var/lib/kubelet/pods/b1afc788-08f2-4260-af9e-5b3a537133ef/volumes" Nov 02 09:58:49.333673 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:49.333660 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" path="/var/lib/kubelet/pods/b78bbcb8-1829-4f6a-8738-9c69969fb66b/volumes" Nov 02 09:58:49.361083 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:49.361039 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9e7156-fc4c465d6-dw27l" podUID="7f08e396-bdfc-46c6-9403-1134d5750473" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.49:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 09:58:49.656952 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:49.656838 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:50.360181 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:50.360133 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:50.839518 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:50.839476 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" podUID="e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" containerName="payload-processing" containerID="cri-o://2f5138acf45d45c3eb2eab4e8b217ce683d0d16ac1f7dbc86024a8622a202570" gracePeriod=25 Nov 02 09:58:50.988884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:50.988851 4467 generic.go:356] "Generic (PLEG): container finished" podID="e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" containerID="2f5138acf45d45c3eb2eab4e8b217ce683d0d16ac1f7dbc86024a8622a202570" exitCode=0 Nov 02 09:58:50.989089 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:50.988934 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" event={"ID":"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1","Type":"ContainerDied","Data":"2f5138acf45d45c3eb2eab4e8b217ce683d0d16ac1f7dbc86024a8622a202570"} Nov 02 09:58:51.058054 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.058021 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:51.142630 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.142604 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:58:51.190757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.190724 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-provider-credentials\") pod \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " Nov 02 09:58:51.191005 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.190817 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-kube-api-access-wq7cj\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-kube-api-access-wq7cj\") pod \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " Nov 02 09:58:51.191005 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.190846 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-routing-overlay-volume\") pod \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " Nov 02 09:58:51.191184 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.191155 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-plugins-config-volume\") pod \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\" (UID: \"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1\") " Nov 02 09:58:51.191291 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.191265 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-routing-overlay-volume" pod "e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" (UID: "e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:51.191738 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.191706 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-plugins-config-volume" pod "e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" (UID: "e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:51.191823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.191771 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:51.196069 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.196041 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-provider-credentials" pod "e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" (UID: "e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:51.196200 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.196179 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-kube-api-access-wq7cj" pod "e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" (UID: "e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1"). InnerVolumeSpecName "kube-api-access-wq7cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:51.292826 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.292779 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq7cj\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-kube-api-access-wq7cj\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:51.292826 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.292813 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:51.292826 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.292823 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:51.745811 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.745770 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:51.938403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.938320 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:51.997478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.997396 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" Nov 02 09:58:51.997478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.997408 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg" event={"ID":"e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1","Type":"ContainerDied","Data":"a352f6a693beecd2f1d29e098b1ae47aaffae5dc223e422a87b54e3ff484255b"} Nov 02 09:58:51.997478 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:51.997474 4467 scope.go:122] "RemoveContainer" containerID="2f5138acf45d45c3eb2eab4e8b217ce683d0d16ac1f7dbc86024a8622a202570" Nov 02 09:58:52.018130 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.018090 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg"] Nov 02 09:58:52.020696 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.020670 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-dc457r6bg"] Nov 02 09:58:52.406229 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.406192 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:52.654748 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.654713 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs"] Nov 02 09:58:52.655053 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.655023 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" containerID="cri-o://869a83b65740ed8d8d23db3f6fa19ce3a7f8320796661fd77f4bcd1ec013f5df" gracePeriod=30 Nov 02 09:58:52.661931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.660931 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd"] Nov 02 09:58:52.661931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.661144 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" containerID="cri-o://69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd" gracePeriod=30 Nov 02 09:58:52.661931 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.661837 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5"] Nov 02 09:58:52.662172 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.662045 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" containerID="cri-o://a059a13084bbf9dbf00543641843091e48d68dbe851d6b4ee2272f2c95e6c939" gracePeriod=30 Nov 02 09:58:52.764335 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:52.764296 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:53.152228 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:53.152181 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:53.332032 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:53.331997 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" path="/var/lib/kubelet/pods/e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1/volumes" Nov 02 09:58:53.860691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:53.860627 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:54.550347 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:54.550314 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:55.332967 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.332879 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:55.825526 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.825483 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:58:55.956913 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.954316 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:55.978401 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.978358 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79"] Nov 02 09:58:55.979262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979234 4467 cpu_manager.go:414] "containerMap: removing container" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" Nov 02 09:58:55.979262 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979259 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979270 4467 cpu_manager.go:414] "containerMap: removing container" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979276 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979282 4467 cpu_manager.go:414] "containerMap: removing container" podUID="e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" containerName="payload-processing" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979287 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" containerName="payload-processing" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979295 4467 cpu_manager.go:414] "containerMap: removing container" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979301 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979308 4467 cpu_manager.go:414] "containerMap: removing container" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979313 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979324 4467 cpu_manager.go:414] "containerMap: removing container" podUID="7f08e396-bdfc-46c6-9403-1134d5750473" containerName="maas-api" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979329 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f08e396-bdfc-46c6-9403-1134d5750473" containerName="maas-api" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979347 4467 cpu_manager.go:414] "containerMap: removing container" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979352 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979365 4467 cpu_manager.go:414] "containerMap: removing container" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" Nov 02 09:58:55.979437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979370 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" Nov 02 09:58:55.979849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979458 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="57315ea7-23ff-442b-8302-c909afc112cc" containerName="istio-proxy" Nov 02 09:58:55.979849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979467 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78bbcb8-1829-4f6a-8738-9c69969fb66b" containerName="istio-proxy" Nov 02 09:58:55.979849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979473 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a918367-c36f-4816-8674-c33e3beb85ce" containerName="istio-proxy" Nov 02 09:58:55.979849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979478 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c134a0-dfe1-4586-b5cc-86eb9b4dafe1" containerName="payload-processing" Nov 02 09:58:55.979849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979483 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7de95e-9a9d-4f12-b7a2-b0f84902240b" containerName="istio-proxy" Nov 02 09:58:55.979849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979489 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f08e396-bdfc-46c6-9403-1134d5750473" containerName="maas-api" Nov 02 09:58:55.979849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979496 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aed7e46-1726-4cbc-bac5-45a69cbce271" containerName="istio-proxy" Nov 02 09:58:55.979849 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.979503 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1afc788-08f2-4260-af9e-5b3a537133ef" containerName="istio-proxy" Nov 02 09:58:55.984528 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.984506 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:55.996149 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:55.996108 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79"] Nov 02 09:58:56.069754 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.069712 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.069975 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.069945 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.070150 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.070123 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqq58\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-kube-api-access-kqq58\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.070238 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.070168 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.138744 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.138662 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:56.171730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.171699 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqq58\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-kube-api-access-kqq58\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.171949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.171738 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.171949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.171774 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.171949 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.171899 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.172910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.172619 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.172910 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.172651 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.175069 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.175041 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.190499 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.190462 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqq58\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-kube-api-access-kqq58\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.306534 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.306495 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:56.329235 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.329208 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:56.592010 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.591978 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79"] Nov 02 09:58:56.593515 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:58:56.593484 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1cb718_f598_423f_8b67_3e63fad07dad.slice/crio-dbf8b992c3f4d1ee2d561a7a8fa9e7a72d0fb7a07c7f07dbf0446e599b154e7c WatchSource:0}: Error finding container dbf8b992c3f4d1ee2d561a7a8fa9e7a72d0fb7a07c7f07dbf0446e599b154e7c: Status 404 returned error can't find the container with id dbf8b992c3f4d1ee2d561a7a8fa9e7a72d0fb7a07c7f07dbf0446e599b154e7c Nov 02 09:58:56.855520 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:56.855431 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:58:56.972645 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:56.972615 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:57.033713 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.033676 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" event={"ID":"4e1cb718-f598-423f-8b67-3e63fad07dad","Type":"ContainerStarted","Data":"494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c"} Nov 02 09:58:57.033926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.033735 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" event={"ID":"4e1cb718-f598-423f-8b67-3e63fad07dad","Type":"ContainerStarted","Data":"dbf8b992c3f4d1ee2d561a7a8fa9e7a72d0fb7a07c7f07dbf0446e599b154e7c"} Nov 02 09:58:57.033926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.033882 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:58:57.067961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.056340 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" podStartSLOduration=2.056324082 podStartE2EDuration="2.056324082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:57.054571442 +0000 UTC m=+1415.894513436" watchObservedRunningTime="2026-11-02 09:58:57.056324082 +0000 UTC m=+1415.896266076" Nov 02 09:58:57.085071 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:57.085031 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-0381f66eae89: secret "agc-cred-0381f66eae89" not found Nov 02 09:58:57.085071 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:57.085077 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79: secret "agc-cred-0381f66eae89" not found Nov 02 09:58:57.085307 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:57.085141 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials podName:4e1cb718-f598-423f-8b67-3e63fad07dad nodeName:}" failed. No retries permitted until 2026-11-02 09:58:57.585121409 +0000 UTC m=+1416.425063400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" (UID: "4e1cb718-f598-423f-8b67-3e63fad07dad") : secret "agc-cred-0381f66eae89" not found Nov 02 09:58:57.159788 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.159705 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79"] Nov 02 09:58:57.177343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.177301 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:57.198430 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.198401 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn"] Nov 02 09:58:57.202507 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.202487 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.215783 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.215758 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn"] Nov 02 09:58:57.288915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.287746 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9zc\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-kube-api-access-sr9zc\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.288915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.287845 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.288915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.287975 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.288915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.288022 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.389174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.389143 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.389366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.389189 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.389366 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.389311 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9zc\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-kube-api-access-sr9zc\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.389452 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.389383 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.390113 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.390084 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.390113 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.390106 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.391866 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.391838 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.403411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.403387 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9zc\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-kube-api-access-sr9zc\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.518494 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.518349 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:57.591885 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:57.591852 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-0381f66eae89: secret "agc-cred-0381f66eae89" not found Nov 02 09:58:57.591885 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:57.591916 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79: secret "agc-cred-0381f66eae89" not found Nov 02 09:58:57.592633 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:57.591994 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials podName:4e1cb718-f598-423f-8b67-3e63fad07dad nodeName:}" failed. No retries permitted until 2026-11-02 09:58:58.59197339 +0000 UTC m=+1417.431915385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" (UID: "4e1cb718-f598-423f-8b67-3e63fad07dad") : secret "agc-cred-0381f66eae89" not found Nov 02 09:58:57.799385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:57.799356 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn"] Nov 02 09:58:57.828931 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:58:57.828877 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567b1176_f272_448a_b099_31949e611315.slice/crio-9b2129904c07fd5a3d75de5f8848027be92618afbf48799feb2711824dce4f04 WatchSource:0}: Error finding container 9b2129904c07fd5a3d75de5f8848027be92618afbf48799feb2711824dce4f04: Status 404 returned error can't find the container with id 9b2129904c07fd5a3d75de5f8848027be92618afbf48799feb2711824dce4f04 Nov 02 09:58:58.023176 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.023147 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:58:58.027915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.027876 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:58.045274 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.045245 4467 generic.go:356] "Generic (PLEG): container finished" podID="a6628631-e521-471e-ab13-311df7e83a02" containerID="a059a13084bbf9dbf00543641843091e48d68dbe851d6b4ee2272f2c95e6c939" exitCode=0 Nov 02 09:58:58.045427 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.045319 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" event={"ID":"a6628631-e521-471e-ab13-311df7e83a02","Type":"ContainerDied","Data":"a059a13084bbf9dbf00543641843091e48d68dbe851d6b4ee2272f2c95e6c939"} Nov 02 09:58:58.047048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.046740 4467 generic.go:356] "Generic (PLEG): container finished" podID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerID="69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd" exitCode=0 Nov 02 09:58:58.047048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.046855 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" event={"ID":"d5aa532a-03d1-4d48-92c1-3057fab0f606","Type":"ContainerDied","Data":"69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd"} Nov 02 09:58:58.047048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.046901 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" event={"ID":"d5aa532a-03d1-4d48-92c1-3057fab0f606","Type":"ContainerDied","Data":"9af96f865d820a19213f485dbffad69c8153fe9c6399645e11012976c4c34cea"} Nov 02 09:58:58.047048 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.046936 4467 scope.go:122] "RemoveContainer" containerID="69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd" Nov 02 09:58:58.047721 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.047345 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd" Nov 02 09:58:58.050813 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.050794 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" event={"ID":"567b1176-f272-448a-b099-31949e611315","Type":"ContainerStarted","Data":"ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769"} Nov 02 09:58:58.051201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.051012 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" event={"ID":"567b1176-f272-448a-b099-31949e611315","Type":"ContainerStarted","Data":"9b2129904c07fd5a3d75de5f8848027be92618afbf48799feb2711824dce4f04"} Nov 02 09:58:58.051201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.051136 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:58:58.059373 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.059344 4467 generic.go:356] "Generic (PLEG): container finished" podID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerID="869a83b65740ed8d8d23db3f6fa19ce3a7f8320796661fd77f4bcd1ec013f5df" exitCode=0 Nov 02 09:58:58.060247 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.060225 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" event={"ID":"7f2107c6-8058-45a0-b71d-ebc0c908b9f2","Type":"ContainerDied","Data":"869a83b65740ed8d8d23db3f6fa19ce3a7f8320796661fd77f4bcd1ec013f5df"} Nov 02 09:58:58.064435 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.064418 4467 scope.go:122] "RemoveContainer" containerID="69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd" Nov 02 09:58:58.064856 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:58.064834 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd\": container with ID starting with 69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd not found: ID does not exist" containerID="69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd" Nov 02 09:58:58.065094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.065064 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd"} err="failed to get container status \"69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd\": rpc error: code = NotFound desc = could not find container \"69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd\": container with ID starting with 69cd3a971bc64599bdd88d30fb933d4f151471cf656949a34ef759e313cc80bd not found: ID does not exist" Nov 02 09:58:58.084861 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.084795 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" podStartSLOduration=1.084764355 podStartE2EDuration="1.084764355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:58.084415344 +0000 UTC m=+1416.924357339" watchObservedRunningTime="2026-11-02 09:58:58.084764355 +0000 UTC m=+1416.924706350" Nov 02 09:58:58.101738 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.101708 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-envoy\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.101933 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.101811 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-token\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-token\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.101933 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.101842 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-data\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.101933 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.101899 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-certs\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.102133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.101956 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5aa532a-03d1-4d48-92c1-3057fab0f606-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa532a-03d1-4d48-92c1-3057fab0f606-istiod-ca-cert\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.102133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102000 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-podinfo\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.102133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102041 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-socket\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.102133 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102103 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-kube-api-access-nw7z4\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-kube-api-access-nw7z4\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.102353 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102186 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-credential-socket\") pod \"d5aa532a-03d1-4d48-92c1-3057fab0f606\" (UID: \"d5aa532a-03d1-4d48-92c1-3057fab0f606\") " Nov 02 09:58:58.102461 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102436 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-certs" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.102677 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102655 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-credential-socket" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.102937 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102912 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.103034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102957 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.103034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.102978 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aa532a-03d1-4d48-92c1-3057fab0f606-istiod-ca-cert" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:58.103212 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.103194 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-socket" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.103422 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.103394 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-data" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.107673 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.107645 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-token" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:58.107989 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.107957 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-envoy" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.108107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.107998 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-podinfo" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:58.109605 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.109579 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-kube-api-access-nw7z4" pod "d5aa532a-03d1-4d48-92c1-3057fab0f606" (UID: "d5aa532a-03d1-4d48-92c1-3057fab0f606"). InnerVolumeSpecName "kube-api-access-nw7z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:58.133061 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.132797 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:58:58.139068 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.138125 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:58:58.203559 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203529 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-certs\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.203730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203577 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-data\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.203730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203665 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istiod-ca-cert\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.203730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203705 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6628631-e521-471e-ab13-311df7e83a02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6628631-e521-471e-ab13-311df7e83a02-istio-podinfo\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.203928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203747 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-data\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.203928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203791 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-envoy\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.203928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203843 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-podinfo\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.203928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203880 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-certs" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.203928 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203919 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-credential-socket\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.204192 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.203917 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-data" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.204192 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204042 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-token\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-token\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.204192 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204112 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6628631-e521-471e-ab13-311df7e83a02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6628631-e521-471e-ab13-311df7e83a02-istiod-ca-cert\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.204192 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204144 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-kube-api-access-rf5x4\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-kube-api-access-rf5x4\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.204192 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204150 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-credential-socket" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.204192 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204180 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-certs\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.204480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204242 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-istio-token\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-istio-token\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.204480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204267 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-kube-api-access-p5wnz\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-kube-api-access-p5wnz\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.204480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204340 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-socket\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.204480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204366 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-socket\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.204480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204417 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-envoy\") pod \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\" (UID: \"7f2107c6-8058-45a0-b71d-ebc0c908b9f2\") " Nov 02 09:58:58.204480 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204444 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-credential-socket\") pod \"a6628631-e521-471e-ab13-311df7e83a02\" (UID: \"a6628631-e521-471e-ab13-311df7e83a02\") " Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205787 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205820 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205835 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205848 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5aa532a-03d1-4d48-92c1-3057fab0f606-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205863 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205877 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205906 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7z4\" (UniqueName: \"kubernetes.io/projected/d5aa532a-03d1-4d48-92c1-3057fab0f606-kube-api-access-nw7z4\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205920 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205936 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205951 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5aa532a-03d1-4d48-92c1-3057fab0f606-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.205961 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.204746 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istiod-ca-cert" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:58.206779 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205031 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-socket" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.206779 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205285 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-socket" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.206779 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.205472 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-credential-socket" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.206779 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.206118 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6628631-e521-471e-ab13-311df7e83a02-istiod-ca-cert" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:58:58.209681 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.208464 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-certs" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.210021 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.209971 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-data" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.212086 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.212062 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-istio-token" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:58.215809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.215775 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-kube-api-access-p5wnz" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "kube-api-access-p5wnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:58.224323 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.224284 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-token" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:58.224608 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.224567 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6628631-e521-471e-ab13-311df7e83a02-istio-podinfo" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:58.225094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.225065 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-envoy" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.226223 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.226199 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-kube-api-access-rf5x4" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "kube-api-access-rf5x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:58:58.231174 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.231153 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-envoy" pod "a6628631-e521-471e-ab13-311df7e83a02" (UID: "a6628631-e521-471e-ab13-311df7e83a02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 09:58:58.232752 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.232731 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-podinfo" pod "7f2107c6-8058-45a0-b71d-ebc0c908b9f2" (UID: "7f2107c6-8058-45a0-b71d-ebc0c908b9f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 09:58:58.306876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306780 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.306876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306824 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6628631-e521-471e-ab13-311df7e83a02-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.306876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306836 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.306876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306846 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.306876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306855 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.306876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306865 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.306876 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306874 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6628631-e521-471e-ab13-311df7e83a02-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.307258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306883 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5x4\" (UniqueName: \"kubernetes.io/projected/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-kube-api-access-rf5x4\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.307258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306905 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.307258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306913 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.307258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306928 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5wnz\" (UniqueName: \"kubernetes.io/projected/a6628631-e521-471e-ab13-311df7e83a02-kube-api-access-p5wnz\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.307258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306938 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.307258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306946 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.307258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306954 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2107c6-8058-45a0-b71d-ebc0c908b9f2-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.307258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.306962 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6628631-e521-471e-ab13-311df7e83a02-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:58:58.397381 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.392066 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd"] Nov 02 09:58:58.400936 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.400870 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-s52zd"] Nov 02 09:58:58.611111 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:58.611075 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-0381f66eae89: secret "agc-cred-0381f66eae89" not found Nov 02 09:58:58.611111 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:58.611111 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79: secret "agc-cred-0381f66eae89" not found Nov 02 09:58:58.611582 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:58:58.611183 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials podName:4e1cb718-f598-423f-8b67-3e63fad07dad nodeName:}" failed. No retries permitted until 2026-11-02 09:59:00.611167906 +0000 UTC m=+1419.451109894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" (UID: "4e1cb718-f598-423f-8b67-3e63fad07dad") : secret "agc-cred-0381f66eae89" not found Nov 02 09:58:58.959111 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:58.959030 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:58:59.066767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.066739 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" Nov 02 09:58:59.066982 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.066733 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs" event={"ID":"7f2107c6-8058-45a0-b71d-ebc0c908b9f2","Type":"ContainerDied","Data":"0372e5dc6e7843f5cb37366f3e66122dc9525b3674a6155ce1292ce13f32bedc"} Nov 02 09:58:59.066982 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.066937 4467 scope.go:122] "RemoveContainer" containerID="869a83b65740ed8d8d23db3f6fa19ce3a7f8320796661fd77f4bcd1ec013f5df" Nov 02 09:58:59.068385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.068253 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" Nov 02 09:58:59.068385 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.068252 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5" event={"ID":"a6628631-e521-471e-ab13-311df7e83a02","Type":"ContainerDied","Data":"0231b514a326b8c626256a2c8655f6a9b6634d41f01ea468fc5e80aee3144938"} Nov 02 09:58:59.079114 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.079094 4467 scope.go:122] "RemoveContainer" containerID="a059a13084bbf9dbf00543641843091e48d68dbe851d6b4ee2272f2c95e6c939" Nov 02 09:58:59.114854 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.114802 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs"] Nov 02 09:58:59.118433 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.118401 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jsxcs"] Nov 02 09:58:59.131875 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.131841 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5"] Nov 02 09:58:59.137256 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.137231 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-zblc5"] Nov 02 09:58:59.333030 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.333002 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" path="/var/lib/kubelet/pods/7f2107c6-8058-45a0-b71d-ebc0c908b9f2/volumes" Nov 02 09:58:59.333437 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.333425 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6628631-e521-471e-ab13-311df7e83a02" path="/var/lib/kubelet/pods/a6628631-e521-471e-ab13-311df7e83a02/volumes" Nov 02 09:58:59.333809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.333797 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" path="/var/lib/kubelet/pods/d5aa532a-03d1-4d48-92c1-3057fab0f606/volumes" Nov 02 09:58:59.665937 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:58:59.665834 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:00.558064 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:00.558033 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:00.638291 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:00.638258 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-0381f66eae89: secret "agc-cred-0381f66eae89" not found Nov 02 09:59:00.638533 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:00.638384 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79: secret "agc-cred-0381f66eae89" not found Nov 02 09:59:00.638533 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:00.638490 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials podName:4e1cb718-f598-423f-8b67-3e63fad07dad nodeName:}" failed. No retries permitted until 2026-11-02 09:59:04.638469304 +0000 UTC m=+1423.478411295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" (UID: "4e1cb718-f598-423f-8b67-3e63fad07dad") : secret "agc-cred-0381f66eae89" not found Nov 02 09:59:00.795017 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:00.794981 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:01.465971 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:01.465943 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:02.268403 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:02.268373 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:02.957094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:02.957063 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:03.751803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:03.751771 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:04.070084 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.070024 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" podUID="4e1cb718-f598-423f-8b67-3e63fad07dad" containerName="payload-processing" containerID="cri-o://494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c" gracePeriod=25 Nov 02 09:59:04.338463 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.338438 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:59:04.459395 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.459358 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:04.480622 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.480581 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:59:04.491278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.483993 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials\") pod \"4e1cb718-f598-423f-8b67-3e63fad07dad\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " Nov 02 09:59:04.491278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.484037 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-routing-overlay-volume\") pod \"4e1cb718-f598-423f-8b67-3e63fad07dad\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " Nov 02 09:59:04.491278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.484083 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-kube-api-access-kqq58\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-kube-api-access-kqq58\") pod \"4e1cb718-f598-423f-8b67-3e63fad07dad\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " Nov 02 09:59:04.491278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.484180 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-plugins-config-volume\") pod \"4e1cb718-f598-423f-8b67-3e63fad07dad\" (UID: \"4e1cb718-f598-423f-8b67-3e63fad07dad\") " Nov 02 09:59:04.491278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.484976 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-routing-overlay-volume" pod "4e1cb718-f598-423f-8b67-3e63fad07dad" (UID: "4e1cb718-f598-423f-8b67-3e63fad07dad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:04.491278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.490660 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials" pod "4e1cb718-f598-423f-8b67-3e63fad07dad" (UID: "4e1cb718-f598-423f-8b67-3e63fad07dad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:04.492263 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.492214 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-kube-api-access-kqq58" pod "4e1cb718-f598-423f-8b67-3e63fad07dad" (UID: "4e1cb718-f598-423f-8b67-3e63fad07dad"). InnerVolumeSpecName "kube-api-access-kqq58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:04.494229 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.494192 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-plugins-config-volume" pod "4e1cb718-f598-423f-8b67-3e63fad07dad" (UID: "4e1cb718-f598-423f-8b67-3e63fad07dad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:04.585613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.585525 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:04.585613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.585558 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:04.585613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.585572 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqq58\" (UniqueName: \"kubernetes.io/projected/4e1cb718-f598-423f-8b67-3e63fad07dad-kube-api-access-kqq58\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:04.585613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:04.585587 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1cb718-f598-423f-8b67-3e63fad07dad-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:05.111170 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.111067 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" Nov 02 09:59:05.111170 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.111104 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" event={"ID":"4e1cb718-f598-423f-8b67-3e63fad07dad","Type":"ContainerDied","Data":"494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c"} Nov 02 09:59:05.111170 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.111016 4467 generic.go:356] "Generic (PLEG): container finished" podID="4e1cb718-f598-423f-8b67-3e63fad07dad" containerID="494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c" exitCode=0 Nov 02 09:59:05.111170 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.111176 4467 scope.go:122] "RemoveContainer" containerID="494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c" Nov 02 09:59:05.111823 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.111272 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79" event={"ID":"4e1cb718-f598-423f-8b67-3e63fad07dad","Type":"ContainerDied","Data":"dbf8b992c3f4d1ee2d561a7a8fa9e7a72d0fb7a07c7f07dbf0446e599b154e7c"} Nov 02 09:59:05.124944 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.124925 4467 scope.go:122] "RemoveContainer" containerID="494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c" Nov 02 09:59:05.125340 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:05.125315 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c\": container with ID starting with 494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c not found: ID does not exist" containerID="494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c" Nov 02 09:59:05.125476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.125343 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c"} err="failed to get container status \"494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c\": rpc error: code = NotFound desc = could not find container \"494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c\": container with ID starting with 494c16a9de84dcc7b660aeb05b4043209d322b205550a0ad1ed684e6d6e7d80c not found: ID does not exist" Nov 02 09:59:05.140538 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.140505 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79"] Nov 02 09:59:05.145340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.145315 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5b5bslh79"] Nov 02 09:59:05.151109 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.151079 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:05.331767 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.331736 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1cb718-f598-423f-8b67-3e63fad07dad" path="/var/lib/kubelet/pods/4e1cb718-f598-423f-8b67-3e63fad07dad/volumes" Nov 02 09:59:05.848621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:05.848589 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:06.560180 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:06.560149 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:07.282772 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:07.282734 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:08.040041 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:08.040006 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:08.905503 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:08.905467 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:09.072803 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.072770 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:59:09.134216 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.134182 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg"] Nov 02 09:59:09.834726 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.834696 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:09.866455 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.866416 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv"] Nov 02 09:59:09.867444 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867417 4467 cpu_manager.go:414] "containerMap: removing container" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" Nov 02 09:59:09.867620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867586 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" Nov 02 09:59:09.867620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867613 4467 cpu_manager.go:414] "containerMap: removing container" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" Nov 02 09:59:09.867620 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867622 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" Nov 02 09:59:09.867860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867651 4467 cpu_manager.go:414] "containerMap: removing container" podUID="4e1cb718-f598-423f-8b67-3e63fad07dad" containerName="payload-processing" Nov 02 09:59:09.867860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867659 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1cb718-f598-423f-8b67-3e63fad07dad" containerName="payload-processing" Nov 02 09:59:09.867860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867685 4467 cpu_manager.go:414] "containerMap: removing container" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" Nov 02 09:59:09.867860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867694 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" Nov 02 09:59:09.867860 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867851 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5aa532a-03d1-4d48-92c1-3057fab0f606" containerName="istio-proxy" Nov 02 09:59:09.868139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867864 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6628631-e521-471e-ab13-311df7e83a02" containerName="istio-proxy" Nov 02 09:59:09.868139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867877 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1cb718-f598-423f-8b67-3e63fad07dad" containerName="payload-processing" Nov 02 09:59:09.868139 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.867909 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2107c6-8058-45a0-b71d-ebc0c908b9f2" containerName="istio-proxy" Nov 02 09:59:09.893919 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.893864 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv"] Nov 02 09:59:09.894250 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.894231 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:09.965201 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.965160 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:09.965384 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.965210 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68cws\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-kube-api-access-68cws\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:09.965429 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.965384 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:09.965522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:09.965504 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.016250 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.016210 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:10.069417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.068581 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.069417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.068745 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.069417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.068809 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.069417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.068857 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68cws\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-kube-api-access-68cws\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.069804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.069658 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.069804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.069738 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.072678 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.072652 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.083219 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.083161 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68cws\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-kube-api-access-68cws\") pod \"payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.218595 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.218513 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:10.498856 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.498833 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv"] Nov 02 09:59:10.500876 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:59:10.500844 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1f9eea_01d6_4eff_a65c_5135503c5bae.slice/crio-e66b62e5e999085bfb865726790e94022f739b9ed5bcce00cf1910eeeabbb8af WatchSource:0}: Error finding container e66b62e5e999085bfb865726790e94022f739b9ed5bcce00cf1910eeeabbb8af: Status 404 returned error can't find the container with id e66b62e5e999085bfb865726790e94022f739b9ed5bcce00cf1910eeeabbb8af Nov 02 09:59:10.770929 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:10.770827 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:11.151703 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:11.151669 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" event={"ID":"6e1f9eea-01d6-4eff-a65c-5135503c5bae","Type":"ContainerStarted","Data":"77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed"} Nov 02 09:59:11.152129 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:11.151727 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" event={"ID":"6e1f9eea-01d6-4eff-a65c-5135503c5bae","Type":"ContainerStarted","Data":"e66b62e5e999085bfb865726790e94022f739b9ed5bcce00cf1910eeeabbb8af"} Nov 02 09:59:11.152129 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:11.151753 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:11.170613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:11.170383 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" podStartSLOduration=2.170366296 podStartE2EDuration="2.170366296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:59:11.169916493 +0000 UTC m=+1430.009858488" watchObservedRunningTime="2026-11-02 09:59:11.170366296 +0000 UTC m=+1430.010308290" Nov 02 09:59:11.457739 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:11.457653 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:11.633308 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:11.633278 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:11.789343 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:11.789256 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:59:11.817832 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:11.817797 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:12.161915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:12.160298 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:13.073566 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:13.073523 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:13.746855 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:13.746803 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:14.134715 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.134646 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" podUID="e491d8b2-2991-4fab-a419-29ca27fb58b2" containerName="payload-processing" containerID="cri-o://10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42" gracePeriod=25 Nov 02 09:59:14.434368 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.434341 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:59:14.443475 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.443442 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:14.528051 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.528015 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-kube-api-access-pn5dp\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-kube-api-access-pn5dp\") pod \"e491d8b2-2991-4fab-a419-29ca27fb58b2\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " Nov 02 09:59:14.528244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.528116 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-plugins-config-volume\") pod \"e491d8b2-2991-4fab-a419-29ca27fb58b2\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " Nov 02 09:59:14.528354 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.528340 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-provider-credentials\") pod \"e491d8b2-2991-4fab-a419-29ca27fb58b2\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " Nov 02 09:59:14.528387 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.528371 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-routing-overlay-volume\") pod \"e491d8b2-2991-4fab-a419-29ca27fb58b2\" (UID: \"e491d8b2-2991-4fab-a419-29ca27fb58b2\") " Nov 02 09:59:14.528533 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.528505 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-plugins-config-volume" pod "e491d8b2-2991-4fab-a419-29ca27fb58b2" (UID: "e491d8b2-2991-4fab-a419-29ca27fb58b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:14.528745 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.528725 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-routing-overlay-volume" pod "e491d8b2-2991-4fab-a419-29ca27fb58b2" (UID: "e491d8b2-2991-4fab-a419-29ca27fb58b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:14.528827 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.528753 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:14.530340 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.530315 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-kube-api-access-pn5dp" pod "e491d8b2-2991-4fab-a419-29ca27fb58b2" (UID: "e491d8b2-2991-4fab-a419-29ca27fb58b2"). InnerVolumeSpecName "kube-api-access-pn5dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:14.530583 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.530568 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-provider-credentials" pod "e491d8b2-2991-4fab-a419-29ca27fb58b2" (UID: "e491d8b2-2991-4fab-a419-29ca27fb58b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:14.629988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.629950 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:14.629988 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.629981 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e491d8b2-2991-4fab-a419-29ca27fb58b2-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:14.630198 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:14.629997 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn5dp\" (UniqueName: \"kubernetes.io/projected/e491d8b2-2991-4fab-a419-29ca27fb58b2-kube-api-access-pn5dp\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:15.142357 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.142319 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:15.181056 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.181020 4467 generic.go:356] "Generic (PLEG): container finished" podID="e491d8b2-2991-4fab-a419-29ca27fb58b2" containerID="10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42" exitCode=0 Nov 02 09:59:15.181218 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.181084 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" Nov 02 09:59:15.181218 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.181101 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" event={"ID":"e491d8b2-2991-4fab-a419-29ca27fb58b2","Type":"ContainerDied","Data":"10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42"} Nov 02 09:59:15.181218 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.181159 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg" event={"ID":"e491d8b2-2991-4fab-a419-29ca27fb58b2","Type":"ContainerDied","Data":"efcdf33a13a73a178c482496c517a9965adb047f5d61150e4fcf4efeaaedbc72"} Nov 02 09:59:15.181218 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.181181 4467 scope.go:122] "RemoveContainer" containerID="10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42" Nov 02 09:59:15.191115 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.191098 4467 scope.go:122] "RemoveContainer" containerID="10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42" Nov 02 09:59:15.191454 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:15.191432 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42\": container with ID starting with 10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42 not found: ID does not exist" containerID="10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42" Nov 02 09:59:15.191554 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.191466 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42"} err="failed to get container status \"10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42\": rpc error: code = NotFound desc = could not find container \"10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42\": container with ID starting with 10d26161b38ffa79afbc81870b2b8cd5926db2d272de6a0ccba3a48192011b42 not found: ID does not exist" Nov 02 09:59:15.209442 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.209411 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg"] Nov 02 09:59:15.212982 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.212960 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7d55tw8zg"] Nov 02 09:59:15.332257 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.332227 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e491d8b2-2991-4fab-a419-29ca27fb58b2" path="/var/lib/kubelet/pods/e491d8b2-2991-4fab-a419-29ca27fb58b2/volumes" Nov 02 09:59:15.848455 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:15.848413 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:16.544433 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:16.544405 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:17.362918 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:17.362876 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:17.949179 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:17.949147 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:18.656901 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:18.656857 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:19.362217 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:19.362188 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:20.165291 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:20.165258 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:20.959976 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:20.959943 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:21.311830 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:21.311796 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn"] Nov 02 09:59:21.315305 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:21.315276 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv"] Nov 02 09:59:21.318372 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:21.318352 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:21.434979 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:21.434929 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:21.686031 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:21.685953 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:21.866648 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:21.866619 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:22.052522 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.052428 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:22.062068 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.062035 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l"] Nov 02 09:59:22.062567 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.062552 4467 cpu_manager.go:414] "containerMap: removing container" podUID="e491d8b2-2991-4fab-a419-29ca27fb58b2" containerName="payload-processing" Nov 02 09:59:22.062636 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.062567 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e491d8b2-2991-4fab-a419-29ca27fb58b2" containerName="payload-processing" Nov 02 09:59:22.062675 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.062645 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="e491d8b2-2991-4fab-a419-29ca27fb58b2" containerName="payload-processing" Nov 02 09:59:22.066703 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.066677 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.072834 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.072809 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-941f3356d954\"" Nov 02 09:59:22.073099 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.073077 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-lhxcq\"" Nov 02 09:59:22.077800 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.077778 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l"] Nov 02 09:59:22.115811 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.115764 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.116241 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.116208 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.116379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.116301 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.116379 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.116359 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsrk7\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-kube-api-access-wsrk7\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.217258 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.217225 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.217411 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.217322 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.217569 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.217545 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.217753 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.217622 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrk7\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-kube-api-access-wsrk7\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.218017 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.217995 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.218505 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.218481 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.220507 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.220483 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.232515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.232491 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrk7\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-kube-api-access-wsrk7\") pod \"payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.243416 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.243382 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:22.382302 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.382268 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:22.549122 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.549091 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l"] Nov 02 09:59:22.630127 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.630071 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:22.669476 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.669448 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l"] Nov 02 09:59:22.671339 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:59:22.671304 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1461e4f8_36a0_43e5_b071_3c465f1774ce.slice/crio-a277ddee75bd6f4147f3e5ccf7176608cf731999ad3c1436219c2e8a0bb8f7f0 WatchSource:0}: Error finding container a277ddee75bd6f4147f3e5ccf7176608cf731999ad3c1436219c2e8a0bb8f7f0: Status 404 returned error can't find the container with id a277ddee75bd6f4147f3e5ccf7176608cf731999ad3c1436219c2e8a0bb8f7f0 Nov 02 09:59:22.836915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:22.832792 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:23.022506 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:23.022374 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:23.236233 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:23.236199 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" event={"ID":"1461e4f8-36a0-43e5-b071-3c465f1774ce","Type":"ContainerStarted","Data":"7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33"} Nov 02 09:59:23.236592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:23.236254 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" event={"ID":"1461e4f8-36a0-43e5-b071-3c465f1774ce","Type":"ContainerStarted","Data":"a277ddee75bd6f4147f3e5ccf7176608cf731999ad3c1436219c2e8a0bb8f7f0"} Nov 02 09:59:23.236592 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:23.236391 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:23.258417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:23.258364 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" podStartSLOduration=1.258349984 podStartE2EDuration="1.258349984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:59:23.256275273 +0000 UTC m=+1442.096217268" watchObservedRunningTime="2026-11-02 09:59:23.258349984 +0000 UTC m=+1442.098291977" Nov 02 09:59:23.265273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:23.265243 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:23.630089 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:23.630048 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:24.349627 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:24.349590 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:24.480112 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:24.480076 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:59:25.170119 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:25.170089 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:25.857784 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:25.857755 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:26.312689 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.312653 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" podUID="567b1176-f272-448a-b099-31949e611315" containerName="payload-processing" containerID="cri-o://ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769" gracePeriod=25 Nov 02 09:59:26.316172 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.316143 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" podUID="6e1f9eea-01d6-4eff-a65c-5135503c5bae" containerName="payload-processing" containerID="cri-o://77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed" gracePeriod=25 Nov 02 09:59:26.623589 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.623559 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:59:26.630019 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.629979 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:26.638510 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.638458 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:26.774269 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774231 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-plugins-config-volume\") pod \"567b1176-f272-448a-b099-31949e611315\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " Nov 02 09:59:26.774459 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774343 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-provider-credentials\") pod \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " Nov 02 09:59:26.774459 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774393 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-routing-overlay-volume\") pod \"567b1176-f272-448a-b099-31949e611315\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " Nov 02 09:59:26.774459 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774453 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-kube-api-access-sr9zc\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-kube-api-access-sr9zc\") pod \"567b1176-f272-448a-b099-31949e611315\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " Nov 02 09:59:26.774599 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774505 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-provider-credentials\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-provider-credentials\") pod \"567b1176-f272-448a-b099-31949e611315\" (UID: \"567b1176-f272-448a-b099-31949e611315\") " Nov 02 09:59:26.774599 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774553 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-kube-api-access-68cws\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-kube-api-access-68cws\") pod \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " Nov 02 09:59:26.774707 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774656 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-plugins-config-volume\") pod \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " Nov 02 09:59:26.774707 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774682 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-routing-overlay-volume\") pod \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\" (UID: \"6e1f9eea-01d6-4eff-a65c-5135503c5bae\") " Nov 02 09:59:26.774809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774727 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-plugins-config-volume" pod "567b1176-f272-448a-b099-31949e611315" (UID: "567b1176-f272-448a-b099-31949e611315"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:26.774809 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.774780 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-routing-overlay-volume" pod "567b1176-f272-448a-b099-31949e611315" (UID: "567b1176-f272-448a-b099-31949e611315"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:26.776018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.775212 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:26.776018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.775238 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/567b1176-f272-448a-b099-31949e611315-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:26.776018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.775283 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-plugins-config-volume" pod "6e1f9eea-01d6-4eff-a65c-5135503c5bae" (UID: "6e1f9eea-01d6-4eff-a65c-5135503c5bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:26.776018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.775555 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-routing-overlay-volume" pod "6e1f9eea-01d6-4eff-a65c-5135503c5bae" (UID: "6e1f9eea-01d6-4eff-a65c-5135503c5bae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:26.776825 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.776800 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-provider-credentials" pod "6e1f9eea-01d6-4eff-a65c-5135503c5bae" (UID: "6e1f9eea-01d6-4eff-a65c-5135503c5bae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:26.777086 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.777060 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-kube-api-access-68cws" pod "6e1f9eea-01d6-4eff-a65c-5135503c5bae" (UID: "6e1f9eea-01d6-4eff-a65c-5135503c5bae"). InnerVolumeSpecName "kube-api-access-68cws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:26.777086 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.777059 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-provider-credentials" pod "567b1176-f272-448a-b099-31949e611315" (UID: "567b1176-f272-448a-b099-31949e611315"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:26.777225 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.777129 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-kube-api-access-sr9zc" pod "567b1176-f272-448a-b099-31949e611315" (UID: "567b1176-f272-448a-b099-31949e611315"). InnerVolumeSpecName "kube-api-access-sr9zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:26.854795 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:26.854702 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:59:26.876474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.876445 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:26.876474 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.876473 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e1f9eea-01d6-4eff-a65c-5135503c5bae-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:26.876873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.876482 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:26.876873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.876493 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr9zc\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-kube-api-access-sr9zc\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:26.876873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.876502 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/567b1176-f272-448a-b099-31949e611315-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:26.876873 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:26.876510 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68cws\" (UniqueName: \"kubernetes.io/projected/6e1f9eea-01d6-4eff-a65c-5135503c5bae-kube-api-access-68cws\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:27.268666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.268580 4467 generic.go:356] "Generic (PLEG): container finished" podID="6e1f9eea-01d6-4eff-a65c-5135503c5bae" containerID="77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed" exitCode=0 Nov 02 09:59:27.268666 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.268634 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" Nov 02 09:59:27.269018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.268660 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" event={"ID":"6e1f9eea-01d6-4eff-a65c-5135503c5bae","Type":"ContainerDied","Data":"77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed"} Nov 02 09:59:27.269018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.268736 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv" event={"ID":"6e1f9eea-01d6-4eff-a65c-5135503c5bae","Type":"ContainerDied","Data":"e66b62e5e999085bfb865726790e94022f739b9ed5bcce00cf1910eeeabbb8af"} Nov 02 09:59:27.269018 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.268760 4467 scope.go:122] "RemoveContainer" containerID="77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed" Nov 02 09:59:27.270094 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.270073 4467 generic.go:356] "Generic (PLEG): container finished" podID="567b1176-f272-448a-b099-31949e611315" containerID="ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769" exitCode=0 Nov 02 09:59:27.270209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.270132 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" Nov 02 09:59:27.270209 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.270133 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" event={"ID":"567b1176-f272-448a-b099-31949e611315","Type":"ContainerDied","Data":"ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769"} Nov 02 09:59:27.270309 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.270236 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn" event={"ID":"567b1176-f272-448a-b099-31949e611315","Type":"ContainerDied","Data":"9b2129904c07fd5a3d75de5f8848027be92618afbf48799feb2711824dce4f04"} Nov 02 09:59:27.277915 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.277869 4467 scope.go:122] "RemoveContainer" containerID="77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed" Nov 02 09:59:27.278172 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:27.278151 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed\": container with ID starting with 77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed not found: ID does not exist" containerID="77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed" Nov 02 09:59:27.278226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.278179 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed"} err="failed to get container status \"77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed\": rpc error: code = NotFound desc = could not find container \"77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed\": container with ID starting with 77590e615f32110915f425b9fcea5137a9c32378bbdb1d48121f7e3acf3701ed not found: ID does not exist" Nov 02 09:59:27.278226 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.278195 4467 scope.go:122] "RemoveContainer" containerID="ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769" Nov 02 09:59:27.286733 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.286717 4467 scope.go:122] "RemoveContainer" containerID="ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769" Nov 02 09:59:27.287017 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:27.286999 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769\": container with ID starting with ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769 not found: ID does not exist" containerID="ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769" Nov 02 09:59:27.287093 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.287020 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769"} err="failed to get container status \"ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769\": rpc error: code = NotFound desc = could not find container \"ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769\": container with ID starting with ac983f20008dea6b5e7b910171d3f744d4177d18edc675c6f9e60ec0c04ed769 not found: ID does not exist" Nov 02 09:59:27.296630 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.296609 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv"] Nov 02 09:59:27.303732 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.303712 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5f76wnrtv"] Nov 02 09:59:27.314730 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.314708 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn"] Nov 02 09:59:27.319485 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.319467 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-5d5dzs9wn"] Nov 02 09:59:27.332208 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.332171 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567b1176-f272-448a-b099-31949e611315" path="/var/lib/kubelet/pods/567b1176-f272-448a-b099-31949e611315/volumes" Nov 02 09:59:27.332728 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.332710 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1f9eea-01d6-4eff-a65c-5135503c5bae" path="/var/lib/kubelet/pods/6e1f9eea-01d6-4eff-a65c-5135503c5bae/volumes" Nov 02 09:59:27.540980 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:27.540879 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:28.236839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.236795 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" podUID="1461e4f8-36a0-43e5-b071-3c465f1774ce" containerName="payload-processing" containerID="cri-o://7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33" gracePeriod=25 Nov 02 09:59:28.241359 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.241334 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:28.495652 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.495586 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:28.595950 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.595880 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-provider-credentials\") pod \"1461e4f8-36a0-43e5-b071-3c465f1774ce\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " Nov 02 09:59:28.596135 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.595970 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-plugins-config-volume\") pod \"1461e4f8-36a0-43e5-b071-3c465f1774ce\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " Nov 02 09:59:28.596135 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.596020 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-kube-api-access-wsrk7\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-kube-api-access-wsrk7\") pod \"1461e4f8-36a0-43e5-b071-3c465f1774ce\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " Nov 02 09:59:28.596273 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.596194 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-routing-overlay-volume\") pod \"1461e4f8-36a0-43e5-b071-3c465f1774ce\" (UID: \"1461e4f8-36a0-43e5-b071-3c465f1774ce\") " Nov 02 09:59:28.596520 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.596487 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-plugins-config-volume" pod "1461e4f8-36a0-43e5-b071-3c465f1774ce" (UID: "1461e4f8-36a0-43e5-b071-3c465f1774ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:28.596621 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.596520 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-routing-overlay-volume" pod "1461e4f8-36a0-43e5-b071-3c465f1774ce" (UID: "1461e4f8-36a0-43e5-b071-3c465f1774ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 09:59:28.596717 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.596699 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:28.596781 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.596719 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1461e4f8-36a0-43e5-b071-3c465f1774ce-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:28.598191 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.598168 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-kube-api-access-wsrk7" pod "1461e4f8-36a0-43e5-b071-3c465f1774ce" (UID: "1461e4f8-36a0-43e5-b071-3c465f1774ce"). InnerVolumeSpecName "kube-api-access-wsrk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:28.598452 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.598435 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-provider-credentials" pod "1461e4f8-36a0-43e5-b071-3c465f1774ce" (UID: "1461e4f8-36a0-43e5-b071-3c465f1774ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 09:59:28.698514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.698476 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:28.698514 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.698514 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsrk7\" (UniqueName: \"kubernetes.io/projected/1461e4f8-36a0-43e5-b071-3c465f1774ce-kube-api-access-wsrk7\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 09:59:28.939841 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:28.939806 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:29.285088 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.284995 4467 generic.go:356] "Generic (PLEG): container finished" podID="1461e4f8-36a0-43e5-b071-3c465f1774ce" containerID="7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33" exitCode=0 Nov 02 09:59:29.285088 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.285055 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" Nov 02 09:59:29.285088 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.285072 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" event={"ID":"1461e4f8-36a0-43e5-b071-3c465f1774ce","Type":"ContainerDied","Data":"7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33"} Nov 02 09:59:29.285691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.285119 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l" event={"ID":"1461e4f8-36a0-43e5-b071-3c465f1774ce","Type":"ContainerDied","Data":"a277ddee75bd6f4147f3e5ccf7176608cf731999ad3c1436219c2e8a0bb8f7f0"} Nov 02 09:59:29.285691 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.285144 4467 scope.go:122] "RemoveContainer" containerID="7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33" Nov 02 09:59:29.297253 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.297237 4467 scope.go:122] "RemoveContainer" containerID="7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33" Nov 02 09:59:29.297515 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:29.297494 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33\": container with ID starting with 7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33 not found: ID does not exist" containerID="7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33" Nov 02 09:59:29.297571 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.297525 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33"} err="failed to get container status \"7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33\": rpc error: code = NotFound desc = could not find container \"7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33\": container with ID starting with 7eeba0f856dcc48af1b7609dc7ed5c8fab3e33deb2a308b9a42460ddab726c33 not found: ID does not exist" Nov 02 09:59:29.312948 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.312924 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l"] Nov 02 09:59:29.314926 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.314886 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-747dvvr8l"] Nov 02 09:59:29.331494 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.331463 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1461e4f8-36a0-43e5-b071-3c465f1774ce" path="/var/lib/kubelet/pods/1461e4f8-36a0-43e5-b071-3c465f1774ce/volumes" Nov 02 09:59:29.661097 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:29.661067 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:30.124847 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:30.124817 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:30.309128 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:30.309091 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:30.496408 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:30.496326 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:31.153610 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:31.153579 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:31.946755 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:31.946722 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:32.680820 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:32.680789 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:32.883804 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:32.883750 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:33.403322 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:33.403284 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:34.163370 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:34.163330 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:34.977093 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:34.977061 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:35.862563 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:35.862526 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:36.555551 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:36.555518 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:36.979884 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:36.979851 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:37.188864 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:37.188830 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:37.372515 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:37.372483 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:38.040521 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.040480 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:38.235175 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.235135 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:38.421150 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.421114 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:38.473757 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.473718 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6"] Nov 02 09:59:38.474257 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474243 4467 cpu_manager.go:414] "containerMap: removing container" podUID="567b1176-f272-448a-b099-31949e611315" containerName="payload-processing" Nov 02 09:59:38.474326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474260 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567b1176-f272-448a-b099-31949e611315" containerName="payload-processing" Nov 02 09:59:38.474326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474292 4467 cpu_manager.go:414] "containerMap: removing container" podUID="6e1f9eea-01d6-4eff-a65c-5135503c5bae" containerName="payload-processing" Nov 02 09:59:38.474326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474297 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1f9eea-01d6-4eff-a65c-5135503c5bae" containerName="payload-processing" Nov 02 09:59:38.474326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474304 4467 cpu_manager.go:414] "containerMap: removing container" podUID="1461e4f8-36a0-43e5-b071-3c465f1774ce" containerName="payload-processing" Nov 02 09:59:38.474326 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474309 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1461e4f8-36a0-43e5-b071-3c465f1774ce" containerName="payload-processing" Nov 02 09:59:38.474568 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474380 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1f9eea-01d6-4eff-a65c-5135503c5bae" containerName="payload-processing" Nov 02 09:59:38.474568 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474388 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="1461e4f8-36a0-43e5-b071-3c465f1774ce" containerName="payload-processing" Nov 02 09:59:38.474568 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.474397 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="567b1176-f272-448a-b099-31949e611315" containerName="payload-processing" Nov 02 09:59:38.479165 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.479142 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.493243 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.488708 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4b9b97e8e575\"" Nov 02 09:59:38.493603 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.493578 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6"] Nov 02 09:59:38.506392 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.506347 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 09:59:38.506392 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.506353 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 09:59:38.506635 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.506401 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 09:59:38.506635 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.506461 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 09:59:38.509022 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.508990 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-plugins-config-volume\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.509184 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.509135 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-provider-credentials\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.509184 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.509177 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-routing-overlay-volume\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.509322 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.509218 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnw4k\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-kube-api-access-jnw4k\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.610789 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.610738 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnw4k\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-kube-api-access-jnw4k\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.611034 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.610865 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-plugins-config-volume\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.611112 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.611038 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-provider-credentials\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.611166 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.611117 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-routing-overlay-volume\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.612244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.612087 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-routing-overlay-volume\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.612244 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.612167 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-plugins-config-volume\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.690983 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.670293 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-provider-credentials\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.690983 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.676874 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnw4k\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-kube-api-access-jnw4k\") pod \"payload-processing-external-model-5bf4f6c6d4-b44j6\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.690983 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.678569 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:38.796316 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.796278 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:38.887935 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:38.885659 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:39.071107 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:39.071076 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:39.081574 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:39.081549 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6"] Nov 02 09:59:39.083973 ip-10-0-138-234 kubenswrapper[4467]: W1102 09:59:39.083943 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c02551d_5e28_466e_8d31_ecffa8db83b3.slice/crio-47b65919cd3e7fb1d2b9e2b750aaa2d72970463be6078deeabce5dd19943427a WatchSource:0}: Error finding container 47b65919cd3e7fb1d2b9e2b750aaa2d72970463be6078deeabce5dd19943427a: Status 404 returned error can't find the container with id 47b65919cd3e7fb1d2b9e2b750aaa2d72970463be6078deeabce5dd19943427a Nov 02 09:59:39.329622 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:39.329528 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:39.348150 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:39.348113 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" event={"ID":"4c02551d-5e28-466e-8d31-ecffa8db83b3","Type":"ContainerStarted","Data":"761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f"} Nov 02 09:59:39.348339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:39.348168 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" event={"ID":"4c02551d-5e28-466e-8d31-ecffa8db83b3","Type":"ContainerStarted","Data":"47b65919cd3e7fb1d2b9e2b750aaa2d72970463be6078deeabce5dd19943427a"} Nov 02 09:59:39.348339 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:39.348220 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:39.369412 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:39.369356 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" podStartSLOduration=1.369342986 podStartE2EDuration="1.369342986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:59:39.368731594 +0000 UTC m=+1458.208673589" watchObservedRunningTime="2026-11-02 09:59:39.369342986 +0000 UTC m=+1458.209284979" Nov 02 09:59:40.061634 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:40.061602 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:40.567782 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:40.567750 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:40.951690 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:40.951594 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:41.655684 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:41.655654 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 09:59:41.688467 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:41.688413 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:42.413271 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:42.413214 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:43.568923 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:43.568108 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:44.935916 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:44.927373 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:46.490295 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:46.490253 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:47.269278 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:47.269242 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:48.073127 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:48.073094 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:50.106178 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:50.106144 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:50.357498 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:50.357421 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 09:59:50.565378 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:50.565327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:51.255756 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:51.255713 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:51.953006 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:51.952973 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:52.737260 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:52.737207 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:53.458580 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:53.458549 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:54.184481 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:54.184437 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:54.867780 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:54.867745 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:55.605431 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:55.605384 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:56.263613 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:56.263566 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:56.854535 ip-10-0-138-234 kubenswrapper[4467]: E1102 09:59:56.854501 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 09:59:56.957834 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:56.957800 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:57.753266 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:57.753231 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:58.463676 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:58.463637 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:59.158417 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:59.158384 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 09:59:59.976839 ip-10-0-138-234 kubenswrapper[4467]: I1102 09:59:59.976801 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:00.857524 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:00.857481 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:01.756853 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:01.756801 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:02.522790 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:02.522744 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:03.362371 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:03.362304 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:04.162980 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:04.162927 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:04.851053 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:04.851001 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:05.590371 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:05.590340 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:06.346317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:06.346285 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:07.051848 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:07.051810 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:07.756395 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:07.756359 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:08.446264 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:08.446227 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:09.160051 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:09.160009 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:09.855624 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:09.855591 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:10.563749 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:10.563715 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:10.816623 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:10.816510 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6"] Nov 02 10:00:10.857818 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:10.857786 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:11.049511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:11.049476 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:11.359338 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:11.359305 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:11.636723 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:11.636632 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:12.057947 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:12.057919 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:12.758821 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:12.758786 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:12.889457 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:12.889421 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:00:13.019641 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.019551 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:13.226618 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.226567 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:13.633868 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.633838 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm"] Nov 02 10:00:13.635564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.635524 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:13.638297 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.638276 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.651856 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.651830 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bd224a8ca00c\"" Nov 02 10:00:13.652077 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.651844 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-k6zl8\"" Nov 02 10:00:13.655570 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.655543 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm"] Nov 02 10:00:13.778447 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.778405 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkhg\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-kube-api-access-5gkhg\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.779408 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.779146 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-provider-credentials\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.779686 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.779664 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-plugins-config-volume\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.779933 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.779915 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-routing-overlay-volume\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.827827 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.827786 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:13.883161 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.883130 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-plugins-config-volume\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.883367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.883183 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-routing-overlay-volume\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.883367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.883289 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkhg\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-kube-api-access-5gkhg\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.883367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.883335 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-provider-credentials\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.883991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.883933 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-routing-overlay-volume\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.884097 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.884012 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-plugins-config-volume\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.885644 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.885624 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-provider-credentials\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.897762 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.897736 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkhg\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-kube-api-access-5gkhg\") pod \"payload-processing-external-model-6cdf744db-7tzrm\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:13.953194 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:13.953164 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:14.244465 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.244432 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm"] Nov 02 10:00:14.245422 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.245395 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:14.250066 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:00:14.250039 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe72886_137c_435e_8245_5abada72d555.slice/crio-46a6b8ece6ceb20da87952f8133ce4423e97b11a9b0d793fdb6fd64ebd66f0a0 WatchSource:0}: Error finding container 46a6b8ece6ceb20da87952f8133ce4423e97b11a9b0d793fdb6fd64ebd66f0a0: Status 404 returned error can't find the container with id 46a6b8ece6ceb20da87952f8133ce4423e97b11a9b0d793fdb6fd64ebd66f0a0 Nov 02 10:00:14.264768 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.264735 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm"] Nov 02 10:00:14.420611 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.420559 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:14.630281 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.630243 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" event={"ID":"bfe72886-137c-435e-8245-5abada72d555","Type":"ContainerStarted","Data":"dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8"} Nov 02 10:00:14.630489 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.630307 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" event={"ID":"bfe72886-137c-435e-8245-5abada72d555","Type":"ContainerStarted","Data":"46a6b8ece6ceb20da87952f8133ce4423e97b11a9b0d793fdb6fd64ebd66f0a0"} Nov 02 10:00:14.630489 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.630338 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:14.650078 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.650010 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" podStartSLOduration=1.6499902130000001 podStartE2EDuration="1.649990213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:00:14.648858451 +0000 UTC m=+1493.488800457" watchObservedRunningTime="2026-11-02 10:00:14.649990213 +0000 UTC m=+1493.489932208" Nov 02 10:00:14.871692 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:14.871653 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:15.249672 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:15.249635 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:15.484038 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:15.484000 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:15.681337 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:15.681304 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:15.817532 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:15.817469 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" podUID="4c02551d-5e28-466e-8d31-ecffa8db83b3" containerName="payload-processing" containerID="cri-o://761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f" gracePeriod=25 Nov 02 10:00:15.867862 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:15.867822 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:16.111621 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.111593 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 10:00:16.227356 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.227269 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-provider-credentials\") pod \"4c02551d-5e28-466e-8d31-ecffa8db83b3\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " Nov 02 10:00:16.227356 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.227334 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-routing-overlay-volume\") pod \"4c02551d-5e28-466e-8d31-ecffa8db83b3\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " Nov 02 10:00:16.227356 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.227356 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-kube-api-access-jnw4k\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-kube-api-access-jnw4k\") pod \"4c02551d-5e28-466e-8d31-ecffa8db83b3\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " Nov 02 10:00:16.227637 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.227391 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-plugins-config-volume\") pod \"4c02551d-5e28-466e-8d31-ecffa8db83b3\" (UID: \"4c02551d-5e28-466e-8d31-ecffa8db83b3\") " Nov 02 10:00:16.227743 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.227714 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-routing-overlay-volume" pod "4c02551d-5e28-466e-8d31-ecffa8db83b3" (UID: "4c02551d-5e28-466e-8d31-ecffa8db83b3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:00:16.227915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.227872 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-plugins-config-volume" pod "4c02551d-5e28-466e-8d31-ecffa8db83b3" (UID: "4c02551d-5e28-466e-8d31-ecffa8db83b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:00:16.229504 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.229484 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-kube-api-access-jnw4k" pod "4c02551d-5e28-466e-8d31-ecffa8db83b3" (UID: "4c02551d-5e28-466e-8d31-ecffa8db83b3"). InnerVolumeSpecName "kube-api-access-jnw4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:00:16.229571 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.229501 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-provider-credentials" pod "4c02551d-5e28-466e-8d31-ecffa8db83b3" (UID: "4c02551d-5e28-466e-8d31-ecffa8db83b3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:00:16.329294 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.329261 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:16.329294 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.329289 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:16.329294 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.329301 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnw4k\" (UniqueName: \"kubernetes.io/projected/4c02551d-5e28-466e-8d31-ecffa8db83b3-kube-api-access-jnw4k\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:16.329537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.329312 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c02551d-5e28-466e-8d31-ecffa8db83b3-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:16.660397 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.660363 4467 generic.go:356] "Generic (PLEG): container finished" podID="4c02551d-5e28-466e-8d31-ecffa8db83b3" containerID="761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f" exitCode=0 Nov 02 10:00:16.660595 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.660409 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" Nov 02 10:00:16.660595 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.660458 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" event={"ID":"4c02551d-5e28-466e-8d31-ecffa8db83b3","Type":"ContainerDied","Data":"761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f"} Nov 02 10:00:16.660595 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.660508 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6" event={"ID":"4c02551d-5e28-466e-8d31-ecffa8db83b3","Type":"ContainerDied","Data":"47b65919cd3e7fb1d2b9e2b750aaa2d72970463be6078deeabce5dd19943427a"} Nov 02 10:00:16.660595 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.660531 4467 scope.go:122] "RemoveContainer" containerID="761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f" Nov 02 10:00:16.670554 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.670538 4467 scope.go:122] "RemoveContainer" containerID="761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f" Nov 02 10:00:16.670797 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:16.670778 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f\": container with ID starting with 761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f not found: ID does not exist" containerID="761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f" Nov 02 10:00:16.670872 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.670804 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f"} err="failed to get container status \"761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f\": rpc error: code = NotFound desc = could not find container \"761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f\": container with ID starting with 761a7573a184b4a6e5be39801b051420103b87b07bd3a36175fbbb8ef256402f not found: ID does not exist" Nov 02 10:00:16.690130 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.690094 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6"] Nov 02 10:00:16.692169 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.692140 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf4f6c6d4-b44j6"] Nov 02 10:00:16.697602 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.697549 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:16.878034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:16.878003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:17.331761 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:17.331728 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c02551d-5e28-466e-8d31-ecffa8db83b3" path="/var/lib/kubelet/pods/4c02551d-5e28-466e-8d31-ecffa8db83b3/volumes" Nov 02 10:00:17.867842 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:17.867791 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:18.540349 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.540309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:18.725335 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.725288 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:18.975112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.975073 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55b55c785-vrkqt"] Nov 02 10:00:18.975863 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.975844 4467 cpu_manager.go:414] "containerMap: removing container" podUID="4c02551d-5e28-466e-8d31-ecffa8db83b3" containerName="payload-processing" Nov 02 10:00:18.975941 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.975866 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c02551d-5e28-466e-8d31-ecffa8db83b3" containerName="payload-processing" Nov 02 10:00:18.976023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.976011 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c02551d-5e28-466e-8d31-ecffa8db83b3" containerName="payload-processing" Nov 02 10:00:18.981516 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.981496 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:18.987421 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.987395 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c3a9f12818cc\"" Nov 02 10:00:18.989767 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:18.989739 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55b55c785-vrkqt"] Nov 02 10:00:19.024039 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.023978 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:19.164935 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.164859 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.165130 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.165018 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-routing-overlay-volume\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.165130 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.165064 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8g5\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-kube-api-access-zt8g5\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.165130 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.165095 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.206913 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.206063 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:19.266370 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.266282 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.266542 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.266397 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-routing-overlay-volume\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.266542 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.266454 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8g5\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-kube-api-access-zt8g5\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.266542 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.266485 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.267229 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.267201 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-routing-overlay-volume\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.267330 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.267246 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.269722 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.269691 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.286923 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.282788 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8g5\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-kube-api-access-zt8g5\") pod \"payload-processing-external-model-55b55c785-vrkqt\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.299518 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.299487 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.584015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.583981 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55b55c785-vrkqt"] Nov 02 10:00:19.584381 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:00:19.584094 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428aa0ad_50cf_4dcd_b1a1_01cd2a300a0d.slice/crio-5e15f20a92a29d5e31484632c389185c56d91fc994bd17b50cc8fc7450b684b4 WatchSource:0}: Error finding container 5e15f20a92a29d5e31484632c389185c56d91fc994bd17b50cc8fc7450b684b4: Status 404 returned error can't find the container with id 5e15f20a92a29d5e31484632c389185c56d91fc994bd17b50cc8fc7450b684b4 Nov 02 10:00:19.631082 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.631051 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" podUID="bfe72886-137c-435e-8245-5abada72d555" containerName="payload-processing" containerID="cri-o://dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8" gracePeriod=25 Nov 02 10:00:19.705774 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.705739 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" event={"ID":"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d","Type":"ContainerStarted","Data":"683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f"} Nov 02 10:00:19.706021 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.705809 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:19.706021 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.705830 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" event={"ID":"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d","Type":"ContainerStarted","Data":"5e15f20a92a29d5e31484632c389185c56d91fc994bd17b50cc8fc7450b684b4"} Nov 02 10:00:19.727204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.727155 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" podStartSLOduration=1.727136658 podStartE2EDuration="1.727136658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:00:19.724698906 +0000 UTC m=+1498.564640899" watchObservedRunningTime="2026-11-02 10:00:19.727136658 +0000 UTC m=+1498.567078653" Nov 02 10:00:19.828686 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.828657 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55b55c785-vrkqt"] Nov 02 10:00:19.876165 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:19.876065 4467 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 02 10:00:19.876165 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:19.876154 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume podName:428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d nodeName:}" failed. No retries permitted until 2026-11-02 10:00:20.376133168 +0000 UTC m=+1499.216075165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume") pod "payload-processing-external-model-55b55c785-vrkqt" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d") : configmap "payload-processing-external-model-plugins" not found Nov 02 10:00:19.877058 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:19.877033 4467 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c3a9f12818cc: secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:19.877058 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:19.877057 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55b55c785-vrkqt: secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:19.877262 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:19.877103 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials podName:428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d nodeName:}" failed. No retries permitted until 2026-11-02 10:00:20.377086629 +0000 UTC m=+1499.217028621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials") pod "payload-processing-external-model-55b55c785-vrkqt" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d") : secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:19.924639 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.924606 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:19.927638 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:19.927611 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:20.080922 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.078097 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-provider-credentials\") pod \"bfe72886-137c-435e-8245-5abada72d555\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " Nov 02 10:00:20.080922 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.078215 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-kube-api-access-5gkhg\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-kube-api-access-5gkhg\") pod \"bfe72886-137c-435e-8245-5abada72d555\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " Nov 02 10:00:20.080922 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.078269 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-plugins-config-volume\") pod \"bfe72886-137c-435e-8245-5abada72d555\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " Nov 02 10:00:20.080922 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.078420 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-routing-overlay-volume\") pod \"bfe72886-137c-435e-8245-5abada72d555\" (UID: \"bfe72886-137c-435e-8245-5abada72d555\") " Nov 02 10:00:20.080922 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.079149 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-routing-overlay-volume" pod "bfe72886-137c-435e-8245-5abada72d555" (UID: "bfe72886-137c-435e-8245-5abada72d555"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:00:20.080922 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.079462 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-plugins-config-volume" pod "bfe72886-137c-435e-8245-5abada72d555" (UID: "bfe72886-137c-435e-8245-5abada72d555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:00:20.081676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.081258 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-provider-credentials" pod "bfe72886-137c-435e-8245-5abada72d555" (UID: "bfe72886-137c-435e-8245-5abada72d555"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:00:20.082753 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.082728 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-kube-api-access-5gkhg" pod "bfe72886-137c-435e-8245-5abada72d555" (UID: "bfe72886-137c-435e-8245-5abada72d555"). InnerVolumeSpecName "kube-api-access-5gkhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:00:20.105985 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.105939 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:20.179933 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.179825 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:20.179933 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.179857 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gkhg\" (UniqueName: \"kubernetes.io/projected/bfe72886-137c-435e-8245-5abada72d555-kube-api-access-5gkhg\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:20.179933 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.179873 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:20.179933 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.179886 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe72886-137c-435e-8245-5abada72d555-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:20.382014 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:20.381971 4467 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 02 10:00:20.382202 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:20.382049 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume podName:428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d nodeName:}" failed. No retries permitted until 2026-11-02 10:00:21.382035283 +0000 UTC m=+1500.221977256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume") pod "payload-processing-external-model-55b55c785-vrkqt" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d") : configmap "payload-processing-external-model-plugins" not found Nov 02 10:00:20.382202 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:20.382067 4467 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c3a9f12818cc: secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:20.382202 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:20.382088 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55b55c785-vrkqt: secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:20.382202 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:20.382140 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials podName:428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d nodeName:}" failed. No retries permitted until 2026-11-02 10:00:21.382127323 +0000 UTC m=+1500.222069314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials") pod "payload-processing-external-model-55b55c785-vrkqt" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d") : secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:20.658172 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.658138 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:20.713839 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.713798 4467 generic.go:356] "Generic (PLEG): container finished" podID="bfe72886-137c-435e-8245-5abada72d555" containerID="dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8" exitCode=0 Nov 02 10:00:20.714038 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.713849 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" Nov 02 10:00:20.714038 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.713885 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" event={"ID":"bfe72886-137c-435e-8245-5abada72d555","Type":"ContainerDied","Data":"dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8"} Nov 02 10:00:20.714038 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.713968 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm" event={"ID":"bfe72886-137c-435e-8245-5abada72d555","Type":"ContainerDied","Data":"46a6b8ece6ceb20da87952f8133ce4423e97b11a9b0d793fdb6fd64ebd66f0a0"} Nov 02 10:00:20.714038 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.713999 4467 scope.go:122] "RemoveContainer" containerID="dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8" Nov 02 10:00:20.725093 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.725072 4467 scope.go:122] "RemoveContainer" containerID="dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8" Nov 02 10:00:20.725398 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:20.725379 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8\": container with ID starting with dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8 not found: ID does not exist" containerID="dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8" Nov 02 10:00:20.725465 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.725411 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8"} err="failed to get container status \"dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8\": rpc error: code = NotFound desc = could not find container \"dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8\": container with ID starting with dc913f556e82a071364638e40e0c7058d6ef74122feb5a6d97ff9302dc10c4c8 not found: ID does not exist" Nov 02 10:00:20.741393 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.741360 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm"] Nov 02 10:00:20.745407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:20.745379 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-7tzrm"] Nov 02 10:00:21.351728 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:21.351681 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe72886-137c-435e-8245-5abada72d555" path="/var/lib/kubelet/pods/bfe72886-137c-435e-8245-5abada72d555/volumes" Nov 02 10:00:21.364280 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:21.364248 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:21.395129 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:21.395098 4467 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c3a9f12818cc: secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:21.395129 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:21.395133 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55b55c785-vrkqt: secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:21.395379 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:21.395107 4467 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 02 10:00:21.395379 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:21.395192 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials podName:428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d nodeName:}" failed. No retries permitted until 2026-11-02 10:00:23.39517443 +0000 UTC m=+1502.235116408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials") pod "payload-processing-external-model-55b55c785-vrkqt" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d") : secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:21.395379 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:21.395283 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume podName:428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d nodeName:}" failed. No retries permitted until 2026-11-02 10:00:23.395260497 +0000 UTC m=+1502.235202474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume") pod "payload-processing-external-model-55b55c785-vrkqt" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d") : configmap "payload-processing-external-model-plugins" not found Nov 02 10:00:22.058848 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:22.058816 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:22.774563 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:22.774481 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:22.965003 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:22.964968 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:23.198461 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:23.198425 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:23.428755 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:23.428663 4467 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 02 10:00:23.428990 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:23.428776 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume podName:428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d nodeName:}" failed. No retries permitted until 2026-11-02 10:00:27.428740126 +0000 UTC m=+1506.268682114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume") pod "payload-processing-external-model-55b55c785-vrkqt" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d") : configmap "payload-processing-external-model-plugins" not found Nov 02 10:00:23.429567 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:23.429544 4467 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c3a9f12818cc: secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:23.429669 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:23.429569 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55b55c785-vrkqt: secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:23.429669 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:23.429614 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials podName:428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d nodeName:}" failed. No retries permitted until 2026-11-02 10:00:27.429598848 +0000 UTC m=+1506.269540834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials") pod "payload-processing-external-model-55b55c785-vrkqt" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d") : secret "agc-cred-c3a9f12818cc" not found Nov 02 10:00:23.505880 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:23.505783 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:23.769633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:23.769550 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:24.040709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.040613 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:24.224043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.224014 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:24.531317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.531276 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:24.554392 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.554359 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv"] Nov 02 10:00:24.554990 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.554954 4467 cpu_manager.go:414] "containerMap: removing container" podUID="bfe72886-137c-435e-8245-5abada72d555" containerName="payload-processing" Nov 02 10:00:24.554990 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.554990 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe72886-137c-435e-8245-5abada72d555" containerName="payload-processing" Nov 02 10:00:24.555194 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.555156 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe72886-137c-435e-8245-5abada72d555" containerName="payload-processing" Nov 02 10:00:24.562897 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.562868 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.569101 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.569072 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv"] Nov 02 10:00:24.570701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.570676 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-dfa5944304a2\"" Nov 02 10:00:24.570876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.570683 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\"" Nov 02 10:00:24.571001 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.570721 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\"" Nov 02 10:00:24.571001 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.570718 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\"" Nov 02 10:00:24.571001 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.570767 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-6qs8v\"" Nov 02 10:00:24.571001 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.570777 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\"" Nov 02 10:00:24.748078 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.748044 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.748266 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.748094 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7vw\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-kube-api-access-nw7vw\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.748310 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.748277 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.748433 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.748414 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.849340 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.849303 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.849541 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.849414 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.849541 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.849465 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7vw\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-kube-api-access-nw7vw\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.849667 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.849544 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.850125 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.850102 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.850277 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.850257 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.851723 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.851702 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.867546 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.867523 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7vw\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-kube-api-access-nw7vw\") pod \"payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:24.882421 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:24.882388 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:25.133404 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:25.133317 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:25.161243 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:25.161217 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv"] Nov 02 10:00:25.162983 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:00:25.162884 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233a10c1_7f16_4a92_8095_9b3752b5b3dd.slice/crio-3c518803e42ebdfdd3085e2ef6cc026b45b1cda224080ec32dd6150cc3b1039d WatchSource:0}: Error finding container 3c518803e42ebdfdd3085e2ef6cc026b45b1cda224080ec32dd6150cc3b1039d: Status 404 returned error can't find the container with id 3c518803e42ebdfdd3085e2ef6cc026b45b1cda224080ec32dd6150cc3b1039d Nov 02 10:00:25.561056 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:25.561026 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:25.746112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:25.746043 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" event={"ID":"233a10c1-7f16-4a92-8095-9b3752b5b3dd","Type":"ContainerStarted","Data":"c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604"} Nov 02 10:00:25.746112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:25.746111 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" event={"ID":"233a10c1-7f16-4a92-8095-9b3752b5b3dd","Type":"ContainerStarted","Data":"3c518803e42ebdfdd3085e2ef6cc026b45b1cda224080ec32dd6150cc3b1039d"} Nov 02 10:00:25.746366 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:25.746347 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:25.767535 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:25.767475 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" podStartSLOduration=1.7674566189999998 podStartE2EDuration="1.767456619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:00:25.766668495 +0000 UTC m=+1504.606610489" watchObservedRunningTime="2026-11-02 10:00:25.767456619 +0000 UTC m=+1504.607398615" Nov 02 10:00:26.258490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:26.258455 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:26.694012 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:26.693969 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:00:26.721711 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:26.721669 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" podUID="428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" containerName="payload-processing" containerID="cri-o://683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f" gracePeriod=25 Nov 02 10:00:26.859549 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:26.859509 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:00:27.056068 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.056041 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:27.152994 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.152946 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:27.178751 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.178715 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume\") pod \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " Nov 02 10:00:27.179003 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.178768 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-kube-api-access-zt8g5\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-kube-api-access-zt8g5\") pod \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " Nov 02 10:00:27.179003 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.178868 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-routing-overlay-volume\") pod \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " Nov 02 10:00:27.179003 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.178924 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials\") pod \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\" (UID: \"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d\") " Nov 02 10:00:27.179310 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.179279 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume" pod "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:00:27.179310 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.179294 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-routing-overlay-volume" pod "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:00:27.179508 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.179484 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:27.179571 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.179513 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:27.181434 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.181412 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials" pod "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:00:27.181931 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.181877 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-kube-api-access-zt8g5" pod "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" (UID: "428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d"). InnerVolumeSpecName "kube-api-access-zt8g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:00:27.280276 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.280187 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt8g5\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-kube-api-access-zt8g5\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:27.280276 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.280225 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:00:27.757573 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.757539 4467 generic.go:356] "Generic (PLEG): container finished" podID="428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" containerID="683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f" exitCode=0 Nov 02 10:00:27.758094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.757581 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" event={"ID":"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d","Type":"ContainerDied","Data":"683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f"} Nov 02 10:00:27.758094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.757630 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" event={"ID":"428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d","Type":"ContainerDied","Data":"5e15f20a92a29d5e31484632c389185c56d91fc994bd17b50cc8fc7450b684b4"} Nov 02 10:00:27.758094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.757582 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55b55c785-vrkqt" Nov 02 10:00:27.758094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.757656 4467 scope.go:122] "RemoveContainer" containerID="683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f" Nov 02 10:00:27.766857 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.766838 4467 scope.go:122] "RemoveContainer" containerID="683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f" Nov 02 10:00:27.767203 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:27.767179 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f\": container with ID starting with 683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f not found: ID does not exist" containerID="683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f" Nov 02 10:00:27.767283 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.767216 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f"} err="failed to get container status \"683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f\": rpc error: code = NotFound desc = could not find container \"683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f\": container with ID starting with 683eaeb240cfd61658d8ebacbf8282df3dc8df470c7aac276cbe37bfc4ccdc6f not found: ID does not exist" Nov 02 10:00:27.778192 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.778164 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55b55c785-vrkqt"] Nov 02 10:00:27.785970 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.785937 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55b55c785-vrkqt"] Nov 02 10:00:27.859175 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:27.859140 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:28.553135 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:28.553102 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:29.252318 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:29.252282 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:29.331540 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:29.331503 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" path="/var/lib/kubelet/pods/428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d/volumes" Nov 02 10:00:29.948028 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:29.947995 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:30.643116 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:30.643068 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:31.445912 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:31.445856 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:32.147370 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:32.147322 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:32.963147 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:32.963111 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:33.646249 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:33.646207 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:34.369818 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:34.369779 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:35.049803 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:35.049764 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:35.735627 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:35.735577 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:36.536857 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:36.536813 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:36.754115 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:36.754077 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:00:37.248836 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:37.248798 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:37.946106 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:37.946077 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:38.644533 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:38.644497 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:39.341433 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:39.341399 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:40.055924 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:40.055875 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:40.843280 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:40.843249 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:41.552202 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:41.552164 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:42.192476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:42.192437 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:00:42.242207 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:42.242172 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:43.108276 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:43.108238 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:43.841983 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:43.841951 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:44.547053 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:44.547013 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:45.254074 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:45.254036 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:45.959521 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:45.959465 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:46.261823 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.261738 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:46.534274 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.534142 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:46.839679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.839634 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8"] Nov 02 10:00:46.840264 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.840244 4467 cpu_manager.go:414] "containerMap: removing container" podUID="428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" containerName="payload-processing" Nov 02 10:00:46.840376 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.840265 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" containerName="payload-processing" Nov 02 10:00:46.840434 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.840401 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="428aa0ad-50cf-4dcd-b1a1-01cd2a300a0d" containerName="payload-processing" Nov 02 10:00:46.844097 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.844065 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:46.851659 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.851384 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\"" Nov 02 10:00:46.851659 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.851442 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\"" Nov 02 10:00:46.852599 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.852479 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\"" Nov 02 10:00:46.852807 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.852789 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\"" Nov 02 10:00:46.853873 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.853850 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-b80a45592980\"" Nov 02 10:00:46.856619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.856595 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8"] Nov 02 10:00:46.873737 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.873697 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:46.909488 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.909448 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9dv\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-kube-api-access-zx9dv\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:46.909694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.909596 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:46.909694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.909662 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:46.909785 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:46.909703 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.012991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.011325 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9dv\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-kube-api-access-zx9dv\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.012991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.011449 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.012991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.011541 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.012991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.011587 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.012991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.012367 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.014301 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.014275 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.016736 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.016710 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.026041 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.026011 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9dv\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-kube-api-access-zx9dv\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.058989 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.058882 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:47.161371 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.161271 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.442302 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.442273 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8"] Nov 02 10:00:47.444707 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:00:47.444671 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb060e7_77e1_4549_8dd2_d807a4b6d919.slice/crio-566edf97b4947f729fb27a4ee9e5db07b4afcca077038b146245e32bbd72ec69 WatchSource:0}: Error finding container 566edf97b4947f729fb27a4ee9e5db07b4afcca077038b146245e32bbd72ec69: Status 404 returned error can't find the container with id 566edf97b4947f729fb27a4ee9e5db07b4afcca077038b146245e32bbd72ec69 Nov 02 10:00:47.741176 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.741089 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:47.883721 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.883680 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" event={"ID":"1cb060e7-77e1-4549-8dd2-d807a4b6d919","Type":"ContainerStarted","Data":"a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5"} Nov 02 10:00:47.883951 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.883749 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" event={"ID":"1cb060e7-77e1-4549-8dd2-d807a4b6d919","Type":"ContainerStarted","Data":"566edf97b4947f729fb27a4ee9e5db07b4afcca077038b146245e32bbd72ec69"} Nov 02 10:00:47.883951 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.883783 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:47.908337 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:47.908278 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" podStartSLOduration=1.908244073 podStartE2EDuration="1.908244073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:00:47.907515678 +0000 UTC m=+1526.747457694" watchObservedRunningTime="2026-11-02 10:00:47.908244073 +0000 UTC m=+1526.748186138" Nov 02 10:00:48.549514 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:48.549473 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:49.261748 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:49.261718 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:49.279824 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:49.279787 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:00:50.109062 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:50.109005 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:50.857226 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:50.857188 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:51.555219 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:51.555188 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:52.258307 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:52.258268 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:52.949013 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:52.948977 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:53.732855 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:53.732816 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:54.567227 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:54.567198 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:55.254092 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:55.254056 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:56.066963 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:56.066904 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:56.803641 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:56.803574 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:56.863497 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:00:56.863457 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:00:57.655794 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:57.655760 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:58.434438 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:58.434409 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:58.894685 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:58.894656 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:00:59.165860 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:59.165777 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:00:59.853346 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:00:59.853306 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:00.645368 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:00.645333 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:01.356770 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:01.356736 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:02.061114 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:02.061082 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:02.757773 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:02.757741 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:03.424336 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:03.424224 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:03.616670 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:03.616627 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:04.272164 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:04.272129 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:04.490742 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:04.490707 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:01:05.050923 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:05.050417 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:05.781511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:05.781473 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:06.438818 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:06.438785 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:07.147318 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:07.147284 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:07.854711 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:07.854674 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:08.474949 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:08.474883 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:08.836353 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:08.836297 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:09.020374 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:09.020325 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:09.637345 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:09.637306 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:10.354469 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:10.354430 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:11.195203 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:11.195157 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:11.472724 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:11.472633 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:01:11.692306 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:11.692273 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:11.986099 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:11.986049 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:12.871075 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:12.871044 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:14.048881 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:14.048838 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:14.765082 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:14.765050 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:15.341192 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:15.341160 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj"] Nov 02 10:01:15.513855 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:15.513814 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:15.696970 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:15.693708 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:16.089597 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:16.089561 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:16.603500 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:16.603468 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:17.575384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:17.575339 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:17.853728 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:17.853629 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8"] Nov 02 10:01:17.988563 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:17.988525 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:18.194131 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:18.194058 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:18.584238 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:18.584198 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:19.359856 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:19.359817 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:20.146966 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:20.146925 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:20.341517 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:20.341475 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" podUID="fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" containerName="payload-processing" containerID="cri-o://017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412" gracePeriod=25 Nov 02 10:01:20.942245 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:20.942211 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:21.836781 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:21.836727 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:22.089897 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:22.089803 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:01:22.476632 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:22.476536 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:22.854407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:22.854361 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" podUID="1cb060e7-77e1-4549-8dd2-d807a4b6d919" containerName="payload-processing" containerID="cri-o://a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5" gracePeriod=25 Nov 02 10:01:22.912633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:22.912600 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:23.118702 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.118643 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:01:23.123224 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.123200 4467 generic.go:356] "Generic (PLEG): container finished" podID="1cb060e7-77e1-4549-8dd2-d807a4b6d919" containerID="a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5" exitCode=0 Nov 02 10:01:23.123368 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.123235 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" Nov 02 10:01:23.123368 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.123243 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" event={"ID":"1cb060e7-77e1-4549-8dd2-d807a4b6d919","Type":"ContainerDied","Data":"a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5"} Nov 02 10:01:23.123368 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.123292 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8" event={"ID":"1cb060e7-77e1-4549-8dd2-d807a4b6d919","Type":"ContainerDied","Data":"566edf97b4947f729fb27a4ee9e5db07b4afcca077038b146245e32bbd72ec69"} Nov 02 10:01:23.123368 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.123322 4467 scope.go:122] "RemoveContainer" containerID="a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5" Nov 02 10:01:23.132367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.132347 4467 scope.go:122] "RemoveContainer" containerID="a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5" Nov 02 10:01:23.132662 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:01:23.132644 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5\": container with ID starting with a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5 not found: ID does not exist" containerID="a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5" Nov 02 10:01:23.132732 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.132676 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5"} err="failed to get container status \"a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5\": rpc error: code = NotFound desc = could not find container \"a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5\": container with ID starting with a65dd20190800abb9483e94268f19127e052b4e4ed43d8ca44f85e58604519c5 not found: ID does not exist" Nov 02 10:01:23.241237 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.241201 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-provider-credentials\") pod \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " Nov 02 10:01:23.241443 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.241314 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-plugins-config-volume\") pod \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " Nov 02 10:01:23.241516 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.241440 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-kube-api-access-zx9dv\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-kube-api-access-zx9dv\") pod \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " Nov 02 10:01:23.241574 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.241512 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-routing-overlay-volume\") pod \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\" (UID: \"1cb060e7-77e1-4549-8dd2-d807a4b6d919\") " Nov 02 10:01:23.241796 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.241765 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-plugins-config-volume" pod "1cb060e7-77e1-4549-8dd2-d807a4b6d919" (UID: "1cb060e7-77e1-4549-8dd2-d807a4b6d919"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:23.241925 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.241854 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-routing-overlay-volume" pod "1cb060e7-77e1-4549-8dd2-d807a4b6d919" (UID: "1cb060e7-77e1-4549-8dd2-d807a4b6d919"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:23.242006 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.241979 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:23.243633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.243604 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-provider-credentials" pod "1cb060e7-77e1-4549-8dd2-d807a4b6d919" (UID: "1cb060e7-77e1-4549-8dd2-d807a4b6d919"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:23.243755 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.243659 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-kube-api-access-zx9dv" pod "1cb060e7-77e1-4549-8dd2-d807a4b6d919" (UID: "1cb060e7-77e1-4549-8dd2-d807a4b6d919"). InnerVolumeSpecName "kube-api-access-zx9dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:23.250180 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.250157 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:23.342674 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.342639 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx9dv\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-kube-api-access-zx9dv\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:23.342674 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.342671 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb060e7-77e1-4549-8dd2-d807a4b6d919-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:23.342871 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.342686 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb060e7-77e1-4549-8dd2-d807a4b6d919-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:23.444333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.444241 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8"] Nov 02 10:01:23.448501 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.448470 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-646b8mwk8"] Nov 02 10:01:23.794537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.794459 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp"] Nov 02 10:01:23.795152 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.795129 4467 cpu_manager.go:414] "containerMap: removing container" podUID="1cb060e7-77e1-4549-8dd2-d807a4b6d919" containerName="payload-processing" Nov 02 10:01:23.795152 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.795149 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb060e7-77e1-4549-8dd2-d807a4b6d919" containerName="payload-processing" Nov 02 10:01:23.795309 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.795240 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb060e7-77e1-4549-8dd2-d807a4b6d919" containerName="payload-processing" Nov 02 10:01:23.799788 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.799768 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.812476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.812449 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8"] Nov 02 10:01:23.831141 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.831104 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp"] Nov 02 10:01:23.831320 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.831214 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.838402 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.838370 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8"] Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849490 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849538 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849582 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849606 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849662 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849686 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849712 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ln4w\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-kube-api-access-8ln4w\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849773 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.849876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.849797 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952173 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952231 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952267 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952296 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952331 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ln4w\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-kube-api-access-8ln4w\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952362 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d5974c-dc84-4465-9561-0be5fd764f9f-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952399 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952438 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952680 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952735 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.952832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952811 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952869 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5xq\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-kube-api-access-hv5xq\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952936 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952969 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.952997 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.953043 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.953068 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.953105 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.953297 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.953439 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.953619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.953530 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.954138 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.953683 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.954474 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.954443 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.959393 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.959368 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.960932 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.960906 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.981695 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.981671 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ln4w\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-kube-api-access-8ln4w\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:23.981858 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:23.981835 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:24.000061 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.000028 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:24.053800 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.053722 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.053800 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.053776 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054054 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.053804 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5xq\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-kube-api-access-hv5xq\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054054 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.053828 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054054 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.053926 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054054 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.053979 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054281 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.054125 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054281 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.054199 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d5974c-dc84-4465-9561-0be5fd764f9f-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054281 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.054231 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054281 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.054239 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.054322 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.054399 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.054484 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.054803 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.054779 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d5974c-dc84-4465-9561-0be5fd764f9f-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.056238 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.056213 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.056395 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.056376 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.067466 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.067444 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.068897 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.068869 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5xq\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-kube-api-access-hv5xq\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.118950 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.118917 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:24.154526 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.154494 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:24.407717 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.407685 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp"] Nov 02 10:01:24.408978 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:01:24.408955 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c3317d_47a5_4eba_8b40_ff3c7837b9e7.slice/crio-1b1ae7340fa1eb1a4dbb0368fef2dc4ecdf8be3e5f2f0a929c93044bc626c421 WatchSource:0}: Error finding container 1b1ae7340fa1eb1a4dbb0368fef2dc4ecdf8be3e5f2f0a929c93044bc626c421: Status 404 returned error can't find the container with id 1b1ae7340fa1eb1a4dbb0368fef2dc4ecdf8be3e5f2f0a929c93044bc626c421 Nov 02 10:01:24.411436 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.411404 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:01:24.411537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.411469 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:01:24.411537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.411503 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:01:24.454498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.454467 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8"] Nov 02 10:01:24.466134 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:01:24.466101 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d5974c_dc84_4465_9561_0be5fd764f9f.slice/crio-21a3f49e9652e6176171880ffc879a0942f690cfaf97d5fa43dab3500940a59f WatchSource:0}: Error finding container 21a3f49e9652e6176171880ffc879a0942f690cfaf97d5fa43dab3500940a59f: Status 404 returned error can't find the container with id 21a3f49e9652e6176171880ffc879a0942f690cfaf97d5fa43dab3500940a59f Nov 02 10:01:24.468622 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.468592 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:01:24.468713 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.468658 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:01:24.468713 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.468686 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:01:24.813053 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.813010 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" podUID="fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:01:24.827937 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:24.824362 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:25.146795 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.146675 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" event={"ID":"c9d5974c-dc84-4465-9561-0be5fd764f9f","Type":"ContainerStarted","Data":"685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6"} Nov 02 10:01:25.147604 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.147562 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" event={"ID":"c9d5974c-dc84-4465-9561-0be5fd764f9f","Type":"ContainerStarted","Data":"21a3f49e9652e6176171880ffc879a0942f690cfaf97d5fa43dab3500940a59f"} Nov 02 10:01:25.152653 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.152620 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" event={"ID":"98c3317d-47a5-4eba-8b40-ff3c7837b9e7","Type":"ContainerStarted","Data":"8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110"} Nov 02 10:01:25.152820 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.152684 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" event={"ID":"98c3317d-47a5-4eba-8b40-ff3c7837b9e7","Type":"ContainerStarted","Data":"1b1ae7340fa1eb1a4dbb0368fef2dc4ecdf8be3e5f2f0a929c93044bc626c421"} Nov 02 10:01:25.155293 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.155272 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:25.157067 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.157039 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 10:01:25.157185 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.157089 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 10:01:25.176154 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.176099 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podStartSLOduration=2.17608497 podStartE2EDuration="2.17608497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:01:25.172094067 +0000 UTC m=+1564.012036061" watchObservedRunningTime="2026-11-02 10:01:25.17608497 +0000 UTC m=+1564.016026965" Nov 02 10:01:25.217199 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.217123 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podStartSLOduration=2.217103439 podStartE2EDuration="2.217103439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:01:25.210387974 +0000 UTC m=+1564.050329970" watchObservedRunningTime="2026-11-02 10:01:25.217103439 +0000 UTC m=+1564.057045434" Nov 02 10:01:25.334538 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.334496 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb060e7-77e1-4549-8dd2-d807a4b6d919" path="/var/lib/kubelet/pods/1cb060e7-77e1-4549-8dd2-d807a4b6d919/volumes" Nov 02 10:01:25.674846 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:25.674805 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:26.119395 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:26.119311 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:26.124396 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:26.124031 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 10:01:26.124396 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:26.124097 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 10:01:26.155581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:26.155547 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 10:01:26.156539 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:26.156147 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 10:01:26.576690 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:26.576652 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:26.867159 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:01:26.867069 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:01:27.129968 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:27.129307 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 10:01:27.129968 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:27.129370 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 10:01:27.155258 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:27.155221 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 10:01:27.155421 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:27.155275 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 10:01:27.401880 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:27.401783 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:28.122513 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:28.122475 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 10:01:28.122841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:28.122795 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 10:01:28.155210 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:28.155179 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 10:01:28.155442 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:28.155419 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 10:01:28.275934 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:28.275880 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:29.107040 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:29.107002 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:29.119515 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:29.119481 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 10:01:29.119838 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:29.119790 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 10:01:29.155165 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:29.155131 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 10:01:29.155398 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:29.155378 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 10:01:29.671721 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:29.671047 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv"] Nov 02 10:01:29.713242 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:29.713198 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:29.908575 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:29.908540 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:30.104732 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:30.104693 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:30.123369 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:30.123330 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 10:01:30.124011 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:30.123976 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 10:01:30.155643 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:30.155604 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 10:01:30.155820 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:30.155657 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 10:01:30.353324 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:30.353289 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:30.859953 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:30.859913 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:31.120292 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:31.120190 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 10:01:31.120292 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:31.120261 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 10:01:31.155547 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:31.155510 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 10:01:31.155956 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:31.155568 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 10:01:31.571330 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:31.571290 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:01:31.758084 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:31.758046 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:32.120701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:32.120658 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 10:01:32.121053 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:32.121009 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 10:01:32.155014 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:32.154974 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 10:01:32.155195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:32.155042 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 10:01:32.534446 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:32.534344 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf"] Nov 02 10:01:32.549399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:32.549349 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" podUID="8534bd6b-43ba-4033-b0e2-051a5f53eb04" containerName="maas-api" containerID="cri-o://4bef1b4449d2e992ff9238996085fbe48c704772cbf32cd564cd323e9f1921d6" gracePeriod=30 Nov 02 10:01:32.555749 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:32.555715 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:33.121731 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.121518 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:33.233108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.233071 4467 generic.go:356] "Generic (PLEG): container finished" podID="8534bd6b-43ba-4033-b0e2-051a5f53eb04" containerID="4bef1b4449d2e992ff9238996085fbe48c704772cbf32cd564cd323e9f1921d6" exitCode=0 Nov 02 10:01:33.233426 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.233401 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" event={"ID":"8534bd6b-43ba-4033-b0e2-051a5f53eb04","Type":"ContainerDied","Data":"4bef1b4449d2e992ff9238996085fbe48c704772cbf32cd564cd323e9f1921d6"} Nov 02 10:01:33.304098 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.302118 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 10:01:33.357538 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.357508 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:33.358158 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.358133 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:33.359550 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.359522 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:33.465015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.462749 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8534bd6b-43ba-4033-b0e2-051a5f53eb04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8534bd6b-43ba-4033-b0e2-051a5f53eb04-openshift-service-ca\") pod \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " Nov 02 10:01:33.465015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.462855 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8534bd6b-43ba-4033-b0e2-051a5f53eb04-kube-api-access-xww25\" (UniqueName: \"kubernetes.io/projected/8534bd6b-43ba-4033-b0e2-051a5f53eb04-kube-api-access-xww25\") pod \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " Nov 02 10:01:33.465015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.463024 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls\") pod \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " Nov 02 10:01:33.465015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.463111 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-metrics-certs\") pod \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\" (UID: \"8534bd6b-43ba-4033-b0e2-051a5f53eb04\") " Nov 02 10:01:33.470117 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.470079 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8534bd6b-43ba-4033-b0e2-051a5f53eb04-openshift-service-ca" pod "8534bd6b-43ba-4033-b0e2-051a5f53eb04" (UID: "8534bd6b-43ba-4033-b0e2-051a5f53eb04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:33.473787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.473733 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-metrics-certs" pod "8534bd6b-43ba-4033-b0e2-051a5f53eb04" (UID: "8534bd6b-43ba-4033-b0e2-051a5f53eb04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:01:33.475255 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.475220 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8534bd6b-43ba-4033-b0e2-051a5f53eb04-kube-api-access-xww25" pod "8534bd6b-43ba-4033-b0e2-051a5f53eb04" (UID: "8534bd6b-43ba-4033-b0e2-051a5f53eb04"). InnerVolumeSpecName "kube-api-access-xww25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:33.476576 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.476533 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls" pod "8534bd6b-43ba-4033-b0e2-051a5f53eb04" (UID: "8534bd6b-43ba-4033-b0e2-051a5f53eb04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:01:33.564803 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.564769 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:33.564803 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.564803 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8534bd6b-43ba-4033-b0e2-051a5f53eb04-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:33.565309 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.564813 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xww25\" (UniqueName: \"kubernetes.io/projected/8534bd6b-43ba-4033-b0e2-051a5f53eb04-kube-api-access-xww25\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:33.565309 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:33.564823 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8534bd6b-43ba-4033-b0e2-051a5f53eb04-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:34.203917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.203472 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:34.214097 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.214049 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:01:34.214279 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.214125 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:01:34.243981 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.243944 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" Nov 02 10:01:34.244636 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.244400 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf" event={"ID":"8534bd6b-43ba-4033-b0e2-051a5f53eb04","Type":"ContainerDied","Data":"070bbb27d69ab8943b2945fe33b7fa038fc5bb1b73372f5b0b20b873c9a57cc4"} Nov 02 10:01:34.244636 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.244499 4467 scope.go:122] "RemoveContainer" containerID="4bef1b4449d2e992ff9238996085fbe48c704772cbf32cd564cd323e9f1921d6" Nov 02 10:01:34.275116 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.275080 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf"] Nov 02 10:01:34.279806 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.279776 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a110ef-5d59847f5b-fgxlf"] Nov 02 10:01:34.671847 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.671799 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" podUID="233a10c1-7f16-4a92-8095-9b3752b5b3dd" containerName="payload-processing" containerID="cri-o://c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604" gracePeriod=25 Nov 02 10:01:34.810515 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:34.810059 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" podUID="fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:01:35.054183 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.054089 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:35.173633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.173600 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:01:35.255164 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.255135 4467 generic.go:356] "Generic (PLEG): container finished" podID="233a10c1-7f16-4a92-8095-9b3752b5b3dd" containerID="c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604" exitCode=0 Nov 02 10:01:35.255422 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.255248 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" Nov 02 10:01:35.255700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.255265 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" event={"ID":"233a10c1-7f16-4a92-8095-9b3752b5b3dd","Type":"ContainerDied","Data":"c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604"} Nov 02 10:01:35.255700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.255600 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv" event={"ID":"233a10c1-7f16-4a92-8095-9b3752b5b3dd","Type":"ContainerDied","Data":"3c518803e42ebdfdd3085e2ef6cc026b45b1cda224080ec32dd6150cc3b1039d"} Nov 02 10:01:35.255700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.255635 4467 scope.go:122] "RemoveContainer" containerID="c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604" Nov 02 10:01:35.258010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.257987 4467 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:01:35.258174 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.258152 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:01:35.281078 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.280947 4467 scope.go:122] "RemoveContainer" containerID="c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604" Nov 02 10:01:35.281456 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:01:35.281429 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604\": container with ID starting with c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604 not found: ID does not exist" containerID="c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604" Nov 02 10:01:35.281565 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.281468 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604"} err="failed to get container status \"c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604\": rpc error: code = NotFound desc = could not find container \"c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604\": container with ID starting with c2609ca8ec1e893633859f780cd87d24a2f3a1479e1519b773c8b4434497c604 not found: ID does not exist" Nov 02 10:01:35.350943 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.350812 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-plugins-config-volume\") pod \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " Nov 02 10:01:35.351157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.351001 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-provider-credentials\") pod \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " Nov 02 10:01:35.351157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.351041 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-routing-overlay-volume\") pod \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " Nov 02 10:01:35.351157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.351086 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-kube-api-access-nw7vw\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-kube-api-access-nw7vw\") pod \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\" (UID: \"233a10c1-7f16-4a92-8095-9b3752b5b3dd\") " Nov 02 10:01:35.351343 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.351311 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-plugins-config-volume" pod "233a10c1-7f16-4a92-8095-9b3752b5b3dd" (UID: "233a10c1-7f16-4a92-8095-9b3752b5b3dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:35.352367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.352329 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-routing-overlay-volume" pod "233a10c1-7f16-4a92-8095-9b3752b5b3dd" (UID: "233a10c1-7f16-4a92-8095-9b3752b5b3dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:35.353461 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.353394 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-provider-credentials" pod "233a10c1-7f16-4a92-8095-9b3752b5b3dd" (UID: "233a10c1-7f16-4a92-8095-9b3752b5b3dd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:35.353461 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.353416 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:35.353684 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.353662 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-kube-api-access-nw7vw" pod "233a10c1-7f16-4a92-8095-9b3752b5b3dd" (UID: "233a10c1-7f16-4a92-8095-9b3752b5b3dd"). InnerVolumeSpecName "kube-api-access-nw7vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:35.354953 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.354928 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8534bd6b-43ba-4033-b0e2-051a5f53eb04" path="/var/lib/kubelet/pods/8534bd6b-43ba-4033-b0e2-051a5f53eb04/volumes" Nov 02 10:01:35.417928 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.417326 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:35.419120 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.419090 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:35.436105 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.436073 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:35.454583 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.454528 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:35.454583 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.454569 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/233a10c1-7f16-4a92-8095-9b3752b5b3dd-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:35.454583 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.454588 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7vw\" (UniqueName: \"kubernetes.io/projected/233a10c1-7f16-4a92-8095-9b3752b5b3dd-kube-api-access-nw7vw\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:35.574330 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.574290 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv"] Nov 02 10:01:35.576620 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.576589 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-699fr9mtv"] Nov 02 10:01:35.773815 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:35.773732 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:36.554208 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:36.554178 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:36.752541 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:36.752512 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:37.340561 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:37.340518 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233a10c1-7f16-4a92-8095-9b3752b5b3dd" path="/var/lib/kubelet/pods/233a10c1-7f16-4a92-8095-9b3752b5b3dd/volumes" Nov 02 10:01:37.387699 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:37.387662 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:37.941459 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:37.941421 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:38.137416 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:38.137378 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:38.324364 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:38.324326 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:38.579694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:38.579607 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:38.815688 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:38.815653 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:39.398732 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:39.398678 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:39.586868 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:39.586834 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:39.985760 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:39.985726 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:40.260723 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.260635 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:40.449371 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.449337 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:40.709287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.709244 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 10:01:40.741314 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.741283 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-plugins-config-volume\") pod \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\" (UID: \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\") " Nov 02 10:01:40.741521 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.741436 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-kube-api-access-jks68\" (UniqueName: \"kubernetes.io/projected/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-kube-api-access-jks68\") pod \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\" (UID: \"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d\") " Nov 02 10:01:40.741746 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.741717 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-plugins-config-volume" pod "fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" (UID: "fd811215-3a20-4fe5-94a3-4c4fe58e5e4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:40.743581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.743547 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-kube-api-access-jks68" pod "fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" (UID: "fd811215-3a20-4fe5-94a3-4c4fe58e5e4d"). InnerVolumeSpecName "kube-api-access-jks68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:40.842351 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.842305 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:40.842351 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:40.842338 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jks68\" (UniqueName: \"kubernetes.io/projected/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d-kube-api-access-jks68\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:41.077377 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.077346 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:41.304436 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.304344 4467 generic.go:356] "Generic (PLEG): container finished" podID="fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" containerID="017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412" exitCode=0 Nov 02 10:01:41.304436 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.304396 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" Nov 02 10:01:41.304675 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.304401 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" event={"ID":"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d","Type":"ContainerDied","Data":"017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412"} Nov 02 10:01:41.304675 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.304538 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj" event={"ID":"fd811215-3a20-4fe5-94a3-4c4fe58e5e4d","Type":"ContainerDied","Data":"5b46c97bbab69f4a7eb641cceef36e4bd968d485a644516ebb3284d6f194d20e"} Nov 02 10:01:41.304675 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.304579 4467 scope.go:122] "RemoveContainer" containerID="017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412" Nov 02 10:01:41.313737 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.313716 4467 scope.go:122] "RemoveContainer" containerID="017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412" Nov 02 10:01:41.314061 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:01:41.314040 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412\": container with ID starting with 017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412 not found: ID does not exist" containerID="017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412" Nov 02 10:01:41.314141 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.314073 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412"} err="failed to get container status \"017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412\": rpc error: code = NotFound desc = could not find container \"017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412\": container with ID starting with 017dcb80f2d77ff32e1b0beefb4642cf26083ae25045d3eab310551b29686412 not found: ID does not exist" Nov 02 10:01:41.338416 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.338379 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj"] Nov 02 10:01:41.349250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.349215 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a110ef-5766965cdf-gd9dj"] Nov 02 10:01:41.371149 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:41.371116 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:01:42.016914 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:42.016512 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:42.655071 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:42.655040 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:43.336282 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:43.336246 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" path="/var/lib/kubelet/pods/fd811215-3a20-4fe5-94a3-4c4fe58e5e4d/volumes" Nov 02 10:01:43.355583 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:43.355554 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:44.056462 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:44.056426 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:44.759960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:44.759928 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:44.964334 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:44.964304 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:45.164452 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:45.164419 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:45.365705 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:45.365669 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:45.555695 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:45.555590 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:45.923785 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:45.923738 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:46.756176 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:46.756141 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:47.714532 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:47.714493 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:48.103506 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:48.103469 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:48.541114 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:48.541026 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:49.246374 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:49.246338 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:49.949072 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:49.949033 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:50.234072 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:50.233991 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:50.503344 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:50.503261 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp"] Nov 02 10:01:50.503716 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:50.503549 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" containerID="cri-o://8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110" gracePeriod=30 Nov 02 10:01:50.505246 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:50.505211 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg"] Nov 02 10:01:50.505657 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:50.505625 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" containerID="cri-o://744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b" gracePeriod=30 Nov 02 10:01:50.508425 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:50.508400 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8"] Nov 02 10:01:50.508645 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:50.508616 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" containerID="cri-o://685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6" gracePeriod=30 Nov 02 10:01:50.616917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:50.616303 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:51.098484 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:51.098446 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:51.859976 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:51.859941 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:52.691005 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:52.690965 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:53.445494 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:53.445456 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:54.145682 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:54.145646 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:54.325805 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:54.325764 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:54.859852 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:54.859823 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:55.549723 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.549691 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:55.783095 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.782996 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:55.806358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.806333 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:55.843349 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.843318 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 10:01:55.963576 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963538 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-podinfo\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.963576 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963579 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-data\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963611 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-token\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-token\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963634 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-data\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963749 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-socket\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963810 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-data\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963846 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-certs\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963858 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-data" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963867 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-data" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963874 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istiod-ca-cert\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.963974 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-token\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-token\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964039 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-envoy\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964081 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-certs\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.964108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964097 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-socket" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964104 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-certs" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964112 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-podinfo\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964179 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-kube-api-access-hv5xq\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-kube-api-access-hv5xq\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964216 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d5974c-dc84-4465-9561-0be5fd764f9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d5974c-dc84-4465-9561-0be5fd764f9f-istiod-ca-cert\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964222 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istiod-ca-cert" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964264 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-envoy\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964299 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-kube-api-access-4xz2q\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-kube-api-access-4xz2q\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964347 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-podinfo\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964443 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-certs\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964445 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-data" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964484 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-kube-api-access-8ln4w\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-kube-api-access-8ln4w\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964522 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-certs" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964529 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-envoy\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964576 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-credential-socket\") pod \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\" (UID: \"bac2dfd4-0623-4aed-b591-3f8334b74cc9\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964631 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istiod-ca-cert\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964691 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-credential-socket\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.964747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964729 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-credential-socket\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964766 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-socket\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964804 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-socket\") pod \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\" (UID: \"98c3317d-47a5-4eba-8b40-ff3c7837b9e7\") " Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.964865 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-token\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-token\") pod \"c9d5974c-dc84-4465-9561-0be5fd764f9f\" (UID: \"c9d5974c-dc84-4465-9561-0be5fd764f9f\") " Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.965379 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.965402 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.965417 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.965432 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.965446 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.965459 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:55.965676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.965473 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:55.966249 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.965959 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d5974c-dc84-4465-9561-0be5fd764f9f-istiod-ca-cert" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:55.966249 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966150 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-credential-socket" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.966364 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966282 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-credential-socket" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.966517 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966413 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istiod-ca-cert" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:01:55.966517 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966482 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-credential-socket" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.966517 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966486 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-socket" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.966756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966549 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-podinfo" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:01:55.966756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966577 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-podinfo" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:01:55.966756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966641 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-certs" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.966756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966711 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-token" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:55.967106 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.966749 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-socket" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.967685 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.967652 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-envoy" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.968070 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.968047 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-token" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:55.968423 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.968373 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-envoy" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.968423 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.968383 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-kube-api-access-4xz2q" pod "bac2dfd4-0623-4aed-b591-3f8334b74cc9" (UID: "bac2dfd4-0623-4aed-b591-3f8334b74cc9"). InnerVolumeSpecName "kube-api-access-4xz2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:55.969058 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.969008 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-kube-api-access-hv5xq" pod "c9d5974c-dc84-4465-9561-0be5fd764f9f" (UID: "c9d5974c-dc84-4465-9561-0be5fd764f9f"). InnerVolumeSpecName "kube-api-access-hv5xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:55.969213 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.969187 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-envoy" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:01:55.969357 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.969294 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-kube-api-access-8ln4w" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "kube-api-access-8ln4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:55.969544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.969518 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-podinfo" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:01:55.969677 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:55.969654 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-token" pod "98c3317d-47a5-4eba-8b40-ff3c7837b9e7" (UID: "98c3317d-47a5-4eba-8b40-ff3c7837b9e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:01:56.066730 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066701 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066730 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066731 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066741 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066750 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5xq\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-kube-api-access-hv5xq\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066759 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9d5974c-dc84-4465-9561-0be5fd764f9f-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066768 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066776 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xz2q\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-kube-api-access-4xz2q\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066785 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066794 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066803 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ln4w\" (UniqueName: \"kubernetes.io/projected/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-kube-api-access-8ln4w\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066811 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066820 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2dfd4-0623-4aed-b591-3f8334b74cc9-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066831 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066839 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066848 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066856 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9d5974c-dc84-4465-9561-0be5fd764f9f-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066864 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c3317d-47a5-4eba-8b40-ff3c7837b9e7-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066872 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9d5974c-dc84-4465-9561-0be5fd764f9f-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066881 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.066960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.066924 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac2dfd4-0623-4aed-b591-3f8334b74cc9-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:01:56.352353 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.352277 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:56.427499 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.427463 4467 generic.go:356] "Generic (PLEG): container finished" podID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerID="685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6" exitCode=0 Nov 02 10:01:56.427680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.427515 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" Nov 02 10:01:56.427680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.427543 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" event={"ID":"c9d5974c-dc84-4465-9561-0be5fd764f9f","Type":"ContainerDied","Data":"685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6"} Nov 02 10:01:56.427680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.427595 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8" event={"ID":"c9d5974c-dc84-4465-9561-0be5fd764f9f","Type":"ContainerDied","Data":"21a3f49e9652e6176171880ffc879a0942f690cfaf97d5fa43dab3500940a59f"} Nov 02 10:01:56.427680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.427620 4467 scope.go:122] "RemoveContainer" containerID="685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6" Nov 02 10:01:56.429176 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.429138 4467 generic.go:356] "Generic (PLEG): container finished" podID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerID="8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110" exitCode=0 Nov 02 10:01:56.429295 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.429195 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" Nov 02 10:01:56.429295 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.429222 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" event={"ID":"98c3317d-47a5-4eba-8b40-ff3c7837b9e7","Type":"ContainerDied","Data":"8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110"} Nov 02 10:01:56.429295 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.429265 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp" event={"ID":"98c3317d-47a5-4eba-8b40-ff3c7837b9e7","Type":"ContainerDied","Data":"1b1ae7340fa1eb1a4dbb0368fef2dc4ecdf8be3e5f2f0a929c93044bc626c421"} Nov 02 10:01:56.430697 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.430677 4467 generic.go:356] "Generic (PLEG): container finished" podID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerID="744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b" exitCode=0 Nov 02 10:01:56.430831 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.430705 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" event={"ID":"bac2dfd4-0623-4aed-b591-3f8334b74cc9","Type":"ContainerDied","Data":"744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b"} Nov 02 10:01:56.430831 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.430744 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" Nov 02 10:01:56.430831 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.430763 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg" event={"ID":"bac2dfd4-0623-4aed-b591-3f8334b74cc9","Type":"ContainerDied","Data":"67c502f7f11499f6df1c757fbf455ca05cdcde7b7bc4eb5609c756dfd27f3c5a"} Nov 02 10:01:56.438144 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.438089 4467 scope.go:122] "RemoveContainer" containerID="685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6" Nov 02 10:01:56.438425 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:01:56.438401 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6\": container with ID starting with 685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6 not found: ID does not exist" containerID="685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6" Nov 02 10:01:56.438485 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.438438 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6"} err="failed to get container status \"685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6\": rpc error: code = NotFound desc = could not find container \"685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6\": container with ID starting with 685014beb50e9f5985892364a82d4cc77e264a7329fd461507d8a1e9518380c6 not found: ID does not exist" Nov 02 10:01:56.438485 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.438460 4467 scope.go:122] "RemoveContainer" containerID="8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110" Nov 02 10:01:56.447715 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.447696 4467 scope.go:122] "RemoveContainer" containerID="8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110" Nov 02 10:01:56.448056 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:01:56.448035 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110\": container with ID starting with 8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110 not found: ID does not exist" containerID="8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110" Nov 02 10:01:56.448113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.448064 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110"} err="failed to get container status \"8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110\": rpc error: code = NotFound desc = could not find container \"8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110\": container with ID starting with 8a0e2dbaaa783621ae3631a97efd07a481ed635f3e85585755c4afac94466110 not found: ID does not exist" Nov 02 10:01:56.448113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.448080 4467 scope.go:122] "RemoveContainer" containerID="744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b" Nov 02 10:01:56.457420 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.457391 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8"] Nov 02 10:01:56.460175 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.460151 4467 scope.go:122] "RemoveContainer" containerID="744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b" Nov 02 10:01:56.460501 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:01:56.460483 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b\": container with ID starting with 744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b not found: ID does not exist" containerID="744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b" Nov 02 10:01:56.460558 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.460507 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b"} err="failed to get container status \"744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b\": rpc error: code = NotFound desc = could not find container \"744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b\": container with ID starting with 744d8ff3fa336d5755b4ade4425c0b3c116f0e0bf5b455f40aaa005214c9af5b not found: ID does not exist" Nov 02 10:01:56.461168 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.461149 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-6fsp8"] Nov 02 10:01:56.473879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.473851 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg"] Nov 02 10:01:56.477997 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.477966 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-rgfsg"] Nov 02 10:01:56.500702 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.500671 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp"] Nov 02 10:01:56.505616 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:56.505589 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p8fzp"] Nov 02 10:01:56.855315 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:01:56.855278 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:01:57.150075 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:57.149989 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:57.332560 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:57.332525 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" path="/var/lib/kubelet/pods/98c3317d-47a5-4eba-8b40-ff3c7837b9e7/volumes" Nov 02 10:01:57.332983 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:57.332969 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" path="/var/lib/kubelet/pods/bac2dfd4-0623-4aed-b591-3f8334b74cc9/volumes" Nov 02 10:01:57.333362 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:57.333349 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" path="/var/lib/kubelet/pods/c9d5974c-dc84-4465-9561-0be5fd764f9f/volumes" Nov 02 10:01:57.861577 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:57.861544 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:58.555666 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:58.555628 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:59.246121 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:59.246087 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:01:59.950589 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:01:59.950554 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:00.653610 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:00.653577 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:01.457103 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:01.457065 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:02.049996 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:02.049961 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:02.950274 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:02.950233 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:03.655917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:03.655848 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:03.798178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:03.798144 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:02:04.535525 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:04.535488 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:05.359692 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:05.359660 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:06.062403 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:06.062367 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:06.760819 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:06.760780 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:07.455214 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:07.455181 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:08.157149 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:08.157112 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:08.854793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:08.854751 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:09.642553 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:09.642508 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:10.380393 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:10.380344 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:11.146809 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:11.146773 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:11.734605 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:11.734556 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:11.906168 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:11.906136 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:12.655813 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:12.655781 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:13.367709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:13.367674 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:14.055566 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:14.055527 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:14.784476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:14.784445 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:15.564273 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:15.564241 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:16.249156 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:16.249124 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:17.151441 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:17.151410 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:17.847234 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:17.847201 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:18.662941 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:18.662858 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:19.355876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:19.355841 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:20.050765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:20.050735 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:20.223773 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:20.223741 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:20.752302 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:20.752211 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:21.093813 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:21.093776 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:02:21.640687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:21.640644 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:22.349459 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:22.349416 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:23.056033 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:23.055994 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:23.756589 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:23.756559 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:24.129848 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:24.129809 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:24.310673 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:24.310641 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:24.654289 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:24.654259 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:25.541050 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:25.541017 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:25.720366 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:25.720324 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:26.210323 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:26.210287 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:02:26.280247 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:26.280211 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:26.963257 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:02:26.963180 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:02:27.212554 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:27.212524 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:27.942348 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:27.942308 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:28.645655 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:28.645614 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:29.545419 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:29.545379 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:30.252193 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:30.252161 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:30.943135 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:30.943054 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:31.646107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:31.646070 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:32.051979 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:32.051878 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:02:32.360327 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:32.360296 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:33.091815 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:33.091772 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:33.848392 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:33.848355 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:34.571303 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:34.571253 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:35.258057 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:35.258021 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:35.956106 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:35.956065 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:36.774010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:36.773976 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:37.565157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:37.565120 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:37.740156 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:37.740106 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:38.273523 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:38.273489 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:38.956647 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:38.956613 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:39.745420 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:39.745382 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:40.445698 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:40.445654 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:41.251811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:41.251779 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:41.968127 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:41.968069 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:42.768765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:42.768733 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:43.389404 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:43.389369 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:02:43.470095 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:43.470063 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:44.254751 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:44.254715 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:45.071763 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:45.071717 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:45.759428 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:45.759375 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:46.562252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:46.562214 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:46.816223 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:46.816122 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:47.015369 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:47.015321 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:47.553426 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:47.553391 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:48.577527 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:48.577491 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:02:48.580311 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:48.580284 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:48.768693 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:48.768663 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:49.578132 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:49.578096 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:50.308731 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:50.308697 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:51.063874 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:51.063831 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:51.862366 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:51.862329 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:52.654845 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:52.654803 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:53.375283 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:53.375247 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:54.391233 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:54.391198 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:54.845298 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:54.845263 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:55.035343 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.035301 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:55.294088 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294001 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx"] Nov 02 10:02:55.294634 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294610 4467 cpu_manager.go:414] "containerMap: removing container" podUID="fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" containerName="payload-processing" Nov 02 10:02:55.294634 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294631 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" containerName="payload-processing" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294653 4467 cpu_manager.go:414] "containerMap: removing container" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294659 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294669 4467 cpu_manager.go:414] "containerMap: removing container" podUID="8534bd6b-43ba-4033-b0e2-051a5f53eb04" containerName="maas-api" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294675 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8534bd6b-43ba-4033-b0e2-051a5f53eb04" containerName="maas-api" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294682 4467 cpu_manager.go:414] "containerMap: removing container" podUID="233a10c1-7f16-4a92-8095-9b3752b5b3dd" containerName="payload-processing" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294686 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233a10c1-7f16-4a92-8095-9b3752b5b3dd" containerName="payload-processing" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294699 4467 cpu_manager.go:414] "containerMap: removing container" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294704 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294719 4467 cpu_manager.go:414] "containerMap: removing container" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294723 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294785 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d5974c-dc84-4465-9561-0be5fd764f9f" containerName="istio-proxy" Nov 02 10:02:55.294793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294794 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac2dfd4-0623-4aed-b591-3f8334b74cc9" containerName="istio-proxy" Nov 02 10:02:55.295198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294803 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd811215-3a20-4fe5-94a3-4c4fe58e5e4d" containerName="payload-processing" Nov 02 10:02:55.295198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294811 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c3317d-47a5-4eba-8b40-ff3c7837b9e7" containerName="istio-proxy" Nov 02 10:02:55.295198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294818 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="8534bd6b-43ba-4033-b0e2-051a5f53eb04" containerName="maas-api" Nov 02 10:02:55.295198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.294823 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="233a10c1-7f16-4a92-8095-9b3752b5b3dd" containerName="payload-processing" Nov 02 10:02:55.298288 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.298265 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.306959 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.306933 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-f77d6b5159d1\"" Nov 02 10:02:55.307128 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.307039 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-d1ad6b3bd8a0\"" Nov 02 10:02:55.307245 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.307046 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-71fd86f9ec46\"" Nov 02 10:02:55.307245 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.307081 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\"" Nov 02 10:02:55.307407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.307109 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-s8xkv\"" Nov 02 10:02:55.307407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.307150 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\"" Nov 02 10:02:55.307407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.307173 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\"" Nov 02 10:02:55.307407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.307133 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\"" Nov 02 10:02:55.307407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.307202 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-407717c0a9b4\"" Nov 02 10:02:55.309678 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.309657 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx"] Nov 02 10:02:55.422153 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.422113 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.422570 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.422246 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.422570 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.422368 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9224\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-kube-api-access-z9224\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.422570 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.422409 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.523701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.523665 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9224\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-kube-api-access-z9224\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.523924 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.523714 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.523924 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.523745 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.523924 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.523794 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.524623 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.524597 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.524745 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.524716 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.526867 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.526842 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.536418 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.536394 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9224\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-kube-api-access-z9224\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.543028 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.543003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:55.615702 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.615669 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:55.625661 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.625632 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j"] Nov 02 10:02:55.629762 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.629739 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.633979 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.633949 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-4c0df7e08666\"" Nov 02 10:02:55.634151 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.634034 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-ebceb8688cb4\"" Nov 02 10:02:55.634151 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.634034 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-6ce0a95c3878\"" Nov 02 10:02:55.634151 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.634105 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-zbbxs\"" Nov 02 10:02:55.637850 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.637786 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\"" Nov 02 10:02:55.640615 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.640126 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\"" Nov 02 10:02:55.640615 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.640199 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\"" Nov 02 10:02:55.640615 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.640254 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\"" Nov 02 10:02:55.642398 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.642373 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j"] Nov 02 10:02:55.725659 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.725620 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbhr\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-kube-api-access-mrbhr\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.725861 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.725805 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.725963 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.725881 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.725963 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.725945 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.734232 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.734204 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-8eb9fb76a028\"" Nov 02 10:02:55.826796 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.826757 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.826796 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.826800 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.827043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.826831 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbhr\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-kube-api-access-mrbhr\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.827043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.827012 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.827726 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.827695 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.827843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.827828 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.829700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.829680 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.838955 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.838925 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbhr\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-kube-api-access-mrbhr\") pod \"payload-processing-external-model-e2e-extmod-a-629129-64cck995j\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:55.917529 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.917501 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx"] Nov 02 10:02:55.920428 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:02:55.920396 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30112bc5_2fef_4629_8944_601ad5b1c87b.slice/crio-66b0a4d57f1dfc6fafd5c6e549362eebeccfa72d0a0b81af2dfb26027a72fa9a WatchSource:0}: Error finding container 66b0a4d57f1dfc6fafd5c6e549362eebeccfa72d0a0b81af2dfb26027a72fa9a: Status 404 returned error can't find the container with id 66b0a4d57f1dfc6fafd5c6e549362eebeccfa72d0a0b81af2dfb26027a72fa9a Nov 02 10:02:55.979988 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:55.979880 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:56.263121 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.263086 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j"] Nov 02 10:02:56.264702 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:02:56.264667 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b086519_b3f1_4ade_9e5a_7996dd032e34.slice/crio-2a0b66c13ede5c41a9484feef250a25c36ac87b0564b9d75b3c6c36746de676d WatchSource:0}: Error finding container 2a0b66c13ede5c41a9484feef250a25c36ac87b0564b9d75b3c6c36746de676d: Status 404 returned error can't find the container with id 2a0b66c13ede5c41a9484feef250a25c36ac87b0564b9d75b3c6c36746de676d Nov 02 10:02:56.355987 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.355955 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:56.773646 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.773610 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:02:56.789676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.789637 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" event={"ID":"5b086519-b3f1-4ade-9e5a-7996dd032e34","Type":"ContainerStarted","Data":"458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19"} Nov 02 10:02:56.789858 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.789731 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" event={"ID":"5b086519-b3f1-4ade-9e5a-7996dd032e34","Type":"ContainerStarted","Data":"2a0b66c13ede5c41a9484feef250a25c36ac87b0564b9d75b3c6c36746de676d"} Nov 02 10:02:56.789858 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.789763 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:02:56.791585 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.791560 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" event={"ID":"30112bc5-2fef-4629-8944-601ad5b1c87b","Type":"ContainerStarted","Data":"0fcf10ae18ee87efdef751a3d8454097b01f24bc94f7e6f408953093592afcc0"} Nov 02 10:02:56.791715 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.791610 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" event={"ID":"30112bc5-2fef-4629-8944-601ad5b1c87b","Type":"ContainerStarted","Data":"66b0a4d57f1dfc6fafd5c6e549362eebeccfa72d0a0b81af2dfb26027a72fa9a"} Nov 02 10:02:56.791765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.791713 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:02:56.810387 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.810326 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" podStartSLOduration=1.810307904 podStartE2EDuration="1.810307904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:02:56.809448781 +0000 UTC m=+1655.649390787" watchObservedRunningTime="2026-11-02 10:02:56.810307904 +0000 UTC m=+1655.650249905" Nov 02 10:02:56.830195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:56.830146 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" podStartSLOduration=1.830129133 podStartE2EDuration="1.830129133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:02:56.827810971 +0000 UTC m=+1655.667752968" watchObservedRunningTime="2026-11-02 10:02:56.830129133 +0000 UTC m=+1655.670071129" Nov 02 10:02:56.856877 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:02:56.856843 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:02:57.076386 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:57.076342 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:57.946070 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:57.946030 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:58.654135 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:58.654103 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:02:59.349810 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:02:59.349773 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:00.044276 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:00.044225 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:00.852672 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:00.852632 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:01.588797 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:01.588755 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:02.438852 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:02.438804 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:03.242382 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:03.242306 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:03.564618 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:03.564579 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:03:03.943792 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:03.943627 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:04.666302 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:04.666267 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:05.428436 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:05.428401 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:06.146000 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:06.145963 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:06.950741 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:06.950703 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:07.751257 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:07.751221 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:07.809993 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:07.809958 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:03:07.810397 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:07.810364 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:03:08.460412 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:08.460375 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:09.147276 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:09.147239 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:09.976027 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:09.975993 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:10.668712 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:10.668676 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:11.345944 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:11.345875 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:11.794246 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:11.794092 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:12.041608 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:12.041564 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:12.843784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:12.843752 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:13.560030 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:13.559984 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:14.250979 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:14.250820 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:14.948926 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:14.948880 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:15.077207 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:15.077174 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:03:15.872798 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:15.872760 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:16.658701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:16.658671 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:17.350975 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:17.350940 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:18.100675 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.100612 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:18.712941 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.712904 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk"] Nov 02 10:03:18.716336 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.716314 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.730355 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.730330 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk"] Nov 02 10:03:18.813602 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.813558 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.813977 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.813951 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.814210 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.814184 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.814378 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.814362 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9199956c-4e75-43c6-8005-402b8d0c2d1f-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.814602 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.814552 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.814828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.814810 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.823199 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.823160 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.823709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.823681 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.823868 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.823852 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxbn\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-kube-api-access-wfxbn\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.860403 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.860369 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:18.925056 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925019 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925232 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925071 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925232 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925129 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925232 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925202 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925232 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925224 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxbn\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-kube-api-access-wfxbn\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925438 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925257 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925438 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925284 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925438 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925312 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925438 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925336 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9199956c-4e75-43c6-8005-402b8d0c2d1f-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925645 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925485 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.925645 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.925563 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.926127 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.926102 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.926127 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.926119 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.926310 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.926290 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9199956c-4e75-43c6-8005-402b8d0c2d1f-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.927942 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.927916 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.928031 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.928015 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.938414 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.938383 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:18.938414 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:18.938403 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxbn\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-kube-api-access-wfxbn\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:19.031557 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.031470 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:19.318778 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.318749 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk"] Nov 02 10:03:19.321415 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:03:19.321385 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9199956c_4e75_43c6_8005_402b8d0c2d1f.slice/crio-94b2224fa05c16b9ebb8b6bc2b025891c0082e278c128802fc12076ddbc5cde5 WatchSource:0}: Error finding container 94b2224fa05c16b9ebb8b6bc2b025891c0082e278c128802fc12076ddbc5cde5: Status 404 returned error can't find the container with id 94b2224fa05c16b9ebb8b6bc2b025891c0082e278c128802fc12076ddbc5cde5 Nov 02 10:03:19.324554 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.324515 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:03:19.324663 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.324605 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:03:19.324663 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.324651 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:03:19.574959 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.574834 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:19.955863 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.955773 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" event={"ID":"9199956c-4e75-43c6-8005-402b8d0c2d1f","Type":"ContainerStarted","Data":"73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20"} Nov 02 10:03:19.955863 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.955830 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" event={"ID":"9199956c-4e75-43c6-8005-402b8d0c2d1f","Type":"ContainerStarted","Data":"94b2224fa05c16b9ebb8b6bc2b025891c0082e278c128802fc12076ddbc5cde5"} Nov 02 10:03:19.981770 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:19.981716 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podStartSLOduration=1.981702611 podStartE2EDuration="1.981702611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:19.978992339 +0000 UTC m=+1678.818934336" watchObservedRunningTime="2026-11-02 10:03:19.981702611 +0000 UTC m=+1678.821644602" Nov 02 10:03:20.031877 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:20.031841 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:20.033607 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:20.033583 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 10:03:20.033754 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:20.033642 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 10:03:20.332459 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:20.332423 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:21.031999 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:21.031961 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 10:03:21.032509 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:21.032011 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 10:03:21.341991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:21.341950 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:22.032345 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:22.032303 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 10:03:22.032767 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:22.032361 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 10:03:22.372778 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:22.372739 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:23.032614 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:23.032574 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 10:03:23.033059 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:23.032630 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 10:03:23.379660 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:23.379624 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:24.032584 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:24.032550 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 10:03:24.032775 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:24.032603 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 10:03:24.554528 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:24.554492 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:25.032722 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:25.032681 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 10:03:25.032915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:25.032752 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 10:03:25.671839 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:25.671800 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:26.036510 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:26.036434 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:26.036850 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:26.036807 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:26.037559 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:26.037537 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:03:26.649349 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:26.649309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:26.856267 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:26.856234 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:03:27.446958 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:27.446922 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:28.157041 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:28.157007 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:28.965432 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:28.965400 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:29.653660 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:29.653625 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:30.064536 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:30.064501 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:03:30.353972 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:30.352363 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:31.279423 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:31.279348 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:31.958808 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:31.958771 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:32.643234 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:32.643198 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:33.428226 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:33.428191 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:34.162792 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:34.162732 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:34.958249 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:34.958212 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:35.651723 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:35.651683 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:36.349317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:36.349278 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:37.050043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:37.050008 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:37.761387 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:37.761335 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:38.549556 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:38.549519 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:39.294374 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:39.294335 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:39.986831 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:39.986786 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:40.404525 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.404491 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:03:40.452823 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.452783 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24"] Nov 02 10:03:40.458773 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.458744 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.466814 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.466777 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24"] Nov 02 10:03:40.497885 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.497809 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.498655 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.498591 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzq2l\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-kube-api-access-zzq2l\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.499009 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.498989 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.499263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.499245 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.512736 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.512686 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:40.600387 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.600345 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.600586 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.600397 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzq2l\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-kube-api-access-zzq2l\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.600586 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.600439 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.600586 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.600464 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.601367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.601336 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.601489 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.601408 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.603307 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.603283 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.613276 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.613240 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzq2l\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-kube-api-access-zzq2l\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796cgqd24\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:40.705537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.705442 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:40.797766 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:40.797726 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:41.099452 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:41.099411 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:41.223517 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:41.208267 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24"] Nov 02 10:03:41.348010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:41.347978 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:41.947972 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:41.947928 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:42.079099 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:42.079056 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" event={"ID":"73965bc2-1f43-4a38-8eca-69919bda0d5f","Type":"ContainerStarted","Data":"5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7"} Nov 02 10:03:42.079316 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:42.079118 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" event={"ID":"73965bc2-1f43-4a38-8eca-69919bda0d5f","Type":"ContainerStarted","Data":"0011576c6ad4add879f2795578cb933941fbcd9bf3a332ed0398a32e5cc9662f"} Nov 02 10:03:42.079316 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:42.079171 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:42.098373 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:42.098324 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" podStartSLOduration=2.09831098 podStartE2EDuration="2.09831098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:42.097562989 +0000 UTC m=+1700.937504984" watchObservedRunningTime="2026-11-02 10:03:42.09831098 +0000 UTC m=+1700.938252974" Nov 02 10:03:42.797999 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:42.797946 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:42.947834 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:42.947799 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24"] Nov 02 10:03:42.989239 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:42.989200 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:43.143215 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:43.143171 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-4c0df7e08666: secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:43.143483 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:43.143469 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24: secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:43.143692 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:43.143678 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials podName:73965bc2-1f43-4a38-8eca-69919bda0d5f nodeName:}" failed. No retries permitted until 2026-11-02 10:03:43.643632717 +0000 UTC m=+1702.483574714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" (UID: "73965bc2-1f43-4a38-8eca-69919bda0d5f") : secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:43.177298 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.177250 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:43.294754 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.294240 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p"] Nov 02 10:03:43.305463 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.305431 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.342187 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.342156 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p"] Nov 02 10:03:43.344933 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.344903 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.345188 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.345163 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwj5\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-kube-api-access-qmwj5\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.345344 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.345327 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.345415 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.345399 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.446787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.446700 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.446787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.446747 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwj5\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-kube-api-access-qmwj5\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.447086 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.446840 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.447086 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.446938 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.447605 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.447579 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.447700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.447631 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.449690 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.449659 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.459590 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.459562 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwj5\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-kube-api-access-qmwj5\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.638980 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.638943 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:43.649913 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:43.649850 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-4c0df7e08666: secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:43.649913 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:43.649904 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24: secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:43.650092 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:43.649992 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials podName:73965bc2-1f43-4a38-8eca-69919bda0d5f nodeName:}" failed. No retries permitted until 2026-11-02 10:03:44.649970033 +0000 UTC m=+1703.489912011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" (UID: "73965bc2-1f43-4a38-8eca-69919bda0d5f") : secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:43.933687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.933661 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p"] Nov 02 10:03:43.935956 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:03:43.935899 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8f0279_4d45_4771_8b75_9b469c39d1d6.slice/crio-6b6189cd0744893073fa5ec560edab09614c8bb50a56b988219e448ae8668e74 WatchSource:0}: Error finding container 6b6189cd0744893073fa5ec560edab09614c8bb50a56b988219e448ae8668e74: Status 404 returned error can't find the container with id 6b6189cd0744893073fa5ec560edab09614c8bb50a56b988219e448ae8668e74 Nov 02 10:03:43.955026 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:43.955000 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:44.092015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:44.091978 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" event={"ID":"cd8f0279-4d45-4771-8b75-9b469c39d1d6","Type":"ContainerStarted","Data":"8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f"} Nov 02 10:03:44.092220 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:44.092041 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" event={"ID":"cd8f0279-4d45-4771-8b75-9b469c39d1d6","Type":"ContainerStarted","Data":"6b6189cd0744893073fa5ec560edab09614c8bb50a56b988219e448ae8668e74"} Nov 02 10:03:44.092220 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:44.092072 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:44.112618 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:44.112567 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" podStartSLOduration=1.112548096 podStartE2EDuration="1.112548096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:44.111030327 +0000 UTC m=+1702.950972323" watchObservedRunningTime="2026-11-02 10:03:44.112548096 +0000 UTC m=+1702.952490088" Nov 02 10:03:44.662391 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:44.662347 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-4c0df7e08666: secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:44.662391 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:44.662389 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24: secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:44.662676 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:44.662456 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials podName:73965bc2-1f43-4a38-8eca-69919bda0d5f nodeName:}" failed. No retries permitted until 2026-11-02 10:03:46.662438295 +0000 UTC m=+1705.502380282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" (UID: "73965bc2-1f43-4a38-8eca-69919bda0d5f") : secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:44.753717 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:44.753665 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:45.551962 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:45.551933 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:46.250958 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:46.250884 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:46.689375 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:46.689344 4467 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-4c0df7e08666: secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:46.689375 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:46.689378 4467 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24: secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:46.689833 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:46.689437 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials podName:73965bc2-1f43-4a38-8eca-69919bda0d5f nodeName:}" failed. No retries permitted until 2026-11-02 10:03:50.689420547 +0000 UTC m=+1709.529362535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" (UID: "73965bc2-1f43-4a38-8eca-69919bda0d5f") : secret "agc-cred-4c0df7e08666" not found Nov 02 10:03:46.949134 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:46.949036 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:47.650627 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:47.650587 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:47.799762 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:47.799725 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp"] Nov 02 10:03:47.800317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:47.800063 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" podUID="69aa7fba-fb2f-45bd-a3ae-ef602853efdd" containerName="discovery" containerID="cri-o://21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4" gracePeriod=30 Nov 02 10:03:48.004534 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.004491 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:03:48.068987 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.068955 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 10:03:48.113056 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.113022 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-local-certs\") pod \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " Nov 02 10:03:48.113056 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.113059 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-kubeconfig\") pod \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " Nov 02 10:03:48.113329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.113106 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-token\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-token\") pod \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " Nov 02 10:03:48.113329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.113126 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-kube-api-access-887bn\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-kube-api-access-887bn\") pod \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " Nov 02 10:03:48.113329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.113185 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-ca-configmap\") pod \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " Nov 02 10:03:48.113329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.113237 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-cacerts\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-cacerts\") pod \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " Nov 02 10:03:48.113557 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.113369 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-dns-cert\") pod \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\" (UID: \"69aa7fba-fb2f-45bd-a3ae-ef602853efdd\") " Nov 02 10:03:48.114913 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.114846 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-ca-configmap" pod "69aa7fba-fb2f-45bd-a3ae-ef602853efdd" (UID: "69aa7fba-fb2f-45bd-a3ae-ef602853efdd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:03:48.118244 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.118029 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-kube-api-access-887bn" pod "69aa7fba-fb2f-45bd-a3ae-ef602853efdd" (UID: "69aa7fba-fb2f-45bd-a3ae-ef602853efdd"). InnerVolumeSpecName "kube-api-access-887bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:03:48.118244 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.118222 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-token" pod "69aa7fba-fb2f-45bd-a3ae-ef602853efdd" (UID: "69aa7fba-fb2f-45bd-a3ae-ef602853efdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:03:48.118581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.118554 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-kubeconfig" pod "69aa7fba-fb2f-45bd-a3ae-ef602853efdd" (UID: "69aa7fba-fb2f-45bd-a3ae-ef602853efdd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:03:48.118750 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.118730 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-cacerts" pod "69aa7fba-fb2f-45bd-a3ae-ef602853efdd" (UID: "69aa7fba-fb2f-45bd-a3ae-ef602853efdd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:03:48.119423 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.119369 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-local-certs" pod "69aa7fba-fb2f-45bd-a3ae-ef602853efdd" (UID: "69aa7fba-fb2f-45bd-a3ae-ef602853efdd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:03:48.120613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.120432 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-dns-cert" pod "69aa7fba-fb2f-45bd-a3ae-ef602853efdd" (UID: "69aa7fba-fb2f-45bd-a3ae-ef602853efdd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:03:48.124014 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.123984 4467 generic.go:356] "Generic (PLEG): container finished" podID="69aa7fba-fb2f-45bd-a3ae-ef602853efdd" containerID="21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4" exitCode=0 Nov 02 10:03:48.124127 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.124054 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" Nov 02 10:03:48.124127 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.124069 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" event={"ID":"69aa7fba-fb2f-45bd-a3ae-ef602853efdd","Type":"ContainerDied","Data":"21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4"} Nov 02 10:03:48.124250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.124142 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp" event={"ID":"69aa7fba-fb2f-45bd-a3ae-ef602853efdd","Type":"ContainerDied","Data":"e5295b09cb94a6850c347cd5a4385c0e4d0e452d7a6c77cb1e2fdd98b5d5deb2"} Nov 02 10:03:48.124250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.124172 4467 scope.go:122] "RemoveContainer" containerID="21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4" Nov 02 10:03:48.145069 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.145041 4467 scope.go:122] "RemoveContainer" containerID="21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4" Nov 02 10:03:48.145403 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:48.145374 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4\": container with ID starting with 21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4 not found: ID does not exist" containerID="21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4" Nov 02 10:03:48.145587 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.145411 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4"} err="failed to get container status \"21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4\": rpc error: code = NotFound desc = could not find container \"21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4\": container with ID starting with 21276251c959eae98194ed8c5edce6fe706c6aae4f1d6a1409ee4d2a19f518c4 not found: ID does not exist" Nov 02 10:03:48.159223 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.159183 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp"] Nov 02 10:03:48.163033 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.163005 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fmbxp"] Nov 02 10:03:48.215601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.215515 4467 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-local-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:48.215601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.215546 4467 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-kubeconfig\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:48.215601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.215557 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:48.215601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.215565 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887bn\" (UniqueName: \"kubernetes.io/projected/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-kube-api-access-887bn\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:48.215601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.215575 4467 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-ca-configmap\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:48.215601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.215584 4467 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-cacerts\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:48.215601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.215592 4467 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/69aa7fba-fb2f-45bd-a3ae-ef602853efdd-istio-csr-dns-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:48.343696 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:48.343658 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:49.047246 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.047213 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:49.092279 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.092228 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" podUID="73965bc2-1f43-4a38-8eca-69919bda0d5f" containerName="payload-processing" containerID="cri-o://5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7" gracePeriod=25 Nov 02 10:03:49.334699 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.334664 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69aa7fba-fb2f-45bd-a3ae-ef602853efdd" path="/var/lib/kubelet/pods/69aa7fba-fb2f-45bd-a3ae-ef602853efdd/volumes" Nov 02 10:03:49.362675 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.362650 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:49.432391 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.432359 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-routing-overlay-volume\") pod \"73965bc2-1f43-4a38-8eca-69919bda0d5f\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " Nov 02 10:03:49.432567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.432453 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-kube-api-access-zzq2l\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-kube-api-access-zzq2l\") pod \"73965bc2-1f43-4a38-8eca-69919bda0d5f\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " Nov 02 10:03:49.432611 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.432587 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials\") pod \"73965bc2-1f43-4a38-8eca-69919bda0d5f\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " Nov 02 10:03:49.432686 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.432669 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-plugins-config-volume\") pod \"73965bc2-1f43-4a38-8eca-69919bda0d5f\" (UID: \"73965bc2-1f43-4a38-8eca-69919bda0d5f\") " Nov 02 10:03:49.432800 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.432780 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-routing-overlay-volume" pod "73965bc2-1f43-4a38-8eca-69919bda0d5f" (UID: "73965bc2-1f43-4a38-8eca-69919bda0d5f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:03:49.433086 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.433064 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:49.433086 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.433079 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-plugins-config-volume" pod "73965bc2-1f43-4a38-8eca-69919bda0d5f" (UID: "73965bc2-1f43-4a38-8eca-69919bda0d5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:03:49.434627 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.434599 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-kube-api-access-zzq2l" pod "73965bc2-1f43-4a38-8eca-69919bda0d5f" (UID: "73965bc2-1f43-4a38-8eca-69919bda0d5f"). InnerVolumeSpecName "kube-api-access-zzq2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:03:49.434734 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.434705 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials" pod "73965bc2-1f43-4a38-8eca-69919bda0d5f" (UID: "73965bc2-1f43-4a38-8eca-69919bda0d5f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:03:49.534443 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.534404 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:49.534443 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.534441 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73965bc2-1f43-4a38-8eca-69919bda0d5f-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:49.534665 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.534454 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzq2l\" (UniqueName: \"kubernetes.io/projected/73965bc2-1f43-4a38-8eca-69919bda0d5f-kube-api-access-zzq2l\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:03:49.748415 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:49.748327 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:50.137500 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.137454 4467 generic.go:356] "Generic (PLEG): container finished" podID="73965bc2-1f43-4a38-8eca-69919bda0d5f" containerID="5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7" exitCode=0 Nov 02 10:03:50.137954 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.137536 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" Nov 02 10:03:50.137954 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.137543 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" event={"ID":"73965bc2-1f43-4a38-8eca-69919bda0d5f","Type":"ContainerDied","Data":"5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7"} Nov 02 10:03:50.137954 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.137601 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24" event={"ID":"73965bc2-1f43-4a38-8eca-69919bda0d5f","Type":"ContainerDied","Data":"0011576c6ad4add879f2795578cb933941fbcd9bf3a332ed0398a32e5cc9662f"} Nov 02 10:03:50.137954 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.137624 4467 scope.go:122] "RemoveContainer" containerID="5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7" Nov 02 10:03:50.147987 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.147965 4467 scope.go:122] "RemoveContainer" containerID="5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7" Nov 02 10:03:50.148303 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:50.148284 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7\": container with ID starting with 5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7 not found: ID does not exist" containerID="5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7" Nov 02 10:03:50.148367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.148311 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7"} err="failed to get container status \"5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7\": rpc error: code = NotFound desc = could not find container \"5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7\": container with ID starting with 5d044cfbcb98bbfc9d7e9f8030cd3d007473e9b50163b6b72c6145e899c23dd7 not found: ID does not exist" Nov 02 10:03:50.160763 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.160732 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24"] Nov 02 10:03:50.164703 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.164679 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796cgqd24"] Nov 02 10:03:50.440657 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:50.440568 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:51.256189 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:51.256149 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:51.333476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:51.333433 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73965bc2-1f43-4a38-8eca-69919bda0d5f" path="/var/lib/kubelet/pods/73965bc2-1f43-4a38-8eca-69919bda0d5f/volumes" Nov 02 10:03:51.954825 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:51.954790 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:52.656356 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:52.656319 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:53.357869 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:53.357838 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:54.129374 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:54.129318 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:54.763804 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:54.763760 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:03:54.959648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:54.959618 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:55.108788 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:55.108757 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:03:55.185917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:55.172201 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx"] Nov 02 10:03:55.469109 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:55.469022 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j"] Nov 02 10:03:55.846674 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:55.846632 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:56.537290 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:56.537258 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:56.855059 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:03:56.855025 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:03:57.247821 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:57.247733 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:57.948252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:57.948219 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:58.665551 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:58.665514 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:03:59.455607 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:03:59.455571 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:00.154033 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.153995 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:00.198620 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.198581 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" podUID="30112bc5-2fef-4629-8944-601ad5b1c87b" containerName="payload-processing" containerID="cri-o://0fcf10ae18ee87efdef751a3d8454097b01f24bc94f7e6f408953093592afcc0" gracePeriod=25 Nov 02 10:04:00.318680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.318647 4467 generic.go:356] "Generic (PLEG): container finished" podID="30112bc5-2fef-4629-8944-601ad5b1c87b" containerID="0fcf10ae18ee87efdef751a3d8454097b01f24bc94f7e6f408953093592afcc0" exitCode=0 Nov 02 10:04:00.318805 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.318718 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" event={"ID":"30112bc5-2fef-4629-8944-601ad5b1c87b","Type":"ContainerDied","Data":"0fcf10ae18ee87efdef751a3d8454097b01f24bc94f7e6f408953093592afcc0"} Nov 02 10:04:00.457301 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.457273 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:04:00.470366 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.470318 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" podUID="5b086519-b3f1-4ade-9e5a-7996dd032e34" containerName="payload-processing" containerID="cri-o://458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19" gracePeriod=25 Nov 02 10:04:00.563748 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.563718 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-kube-api-access-z9224\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-kube-api-access-z9224\") pod \"30112bc5-2fef-4629-8944-601ad5b1c87b\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " Nov 02 10:04:00.563981 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.563758 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-plugins-config-volume\") pod \"30112bc5-2fef-4629-8944-601ad5b1c87b\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " Nov 02 10:04:00.563981 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.563810 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-routing-overlay-volume\") pod \"30112bc5-2fef-4629-8944-601ad5b1c87b\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " Nov 02 10:04:00.563981 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.563846 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-provider-credentials\") pod \"30112bc5-2fef-4629-8944-601ad5b1c87b\" (UID: \"30112bc5-2fef-4629-8944-601ad5b1c87b\") " Nov 02 10:04:00.564285 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.564258 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-plugins-config-volume" pod "30112bc5-2fef-4629-8944-601ad5b1c87b" (UID: "30112bc5-2fef-4629-8944-601ad5b1c87b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:00.564343 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.564261 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-routing-overlay-volume" pod "30112bc5-2fef-4629-8944-601ad5b1c87b" (UID: "30112bc5-2fef-4629-8944-601ad5b1c87b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:00.566066 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.566044 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-provider-credentials" pod "30112bc5-2fef-4629-8944-601ad5b1c87b" (UID: "30112bc5-2fef-4629-8944-601ad5b1c87b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:00.566151 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.566136 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-kube-api-access-z9224" pod "30112bc5-2fef-4629-8944-601ad5b1c87b" (UID: "30112bc5-2fef-4629-8944-601ad5b1c87b"). InnerVolumeSpecName "kube-api-access-z9224". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:00.665875 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.665822 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:00.665875 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.665865 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9224\" (UniqueName: \"kubernetes.io/projected/30112bc5-2fef-4629-8944-601ad5b1c87b-kube-api-access-z9224\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:00.665875 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.665880 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:00.666218 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.665925 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30112bc5-2fef-4629-8944-601ad5b1c87b-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:00.714757 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.714731 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:04:00.841328 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.841292 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:00.867240 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.867209 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-routing-overlay-volume\") pod \"5b086519-b3f1-4ade-9e5a-7996dd032e34\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " Nov 02 10:04:00.867418 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.867332 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-provider-credentials\") pod \"5b086519-b3f1-4ade-9e5a-7996dd032e34\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " Nov 02 10:04:00.867418 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.867358 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-kube-api-access-mrbhr\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-kube-api-access-mrbhr\") pod \"5b086519-b3f1-4ade-9e5a-7996dd032e34\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " Nov 02 10:04:00.867519 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.867416 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-plugins-config-volume\") pod \"5b086519-b3f1-4ade-9e5a-7996dd032e34\" (UID: \"5b086519-b3f1-4ade-9e5a-7996dd032e34\") " Nov 02 10:04:00.867601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.867577 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-routing-overlay-volume" pod "5b086519-b3f1-4ade-9e5a-7996dd032e34" (UID: "5b086519-b3f1-4ade-9e5a-7996dd032e34"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:00.867867 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.867842 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-plugins-config-volume" pod "5b086519-b3f1-4ade-9e5a-7996dd032e34" (UID: "5b086519-b3f1-4ade-9e5a-7996dd032e34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:00.867867 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.867860 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:00.869488 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.869465 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-provider-credentials" pod "5b086519-b3f1-4ade-9e5a-7996dd032e34" (UID: "5b086519-b3f1-4ade-9e5a-7996dd032e34"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:00.869599 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.869489 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-kube-api-access-mrbhr" pod "5b086519-b3f1-4ade-9e5a-7996dd032e34" (UID: "5b086519-b3f1-4ade-9e5a-7996dd032e34"). InnerVolumeSpecName "kube-api-access-mrbhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:00.969358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.969267 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbhr\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-kube-api-access-mrbhr\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:00.969358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.969300 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b086519-b3f1-4ade-9e5a-7996dd032e34-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:00.969358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:00.969311 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b086519-b3f1-4ade-9e5a-7996dd032e34-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:01.325274 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.325227 4467 generic.go:356] "Generic (PLEG): container finished" podID="5b086519-b3f1-4ade-9e5a-7996dd032e34" containerID="458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19" exitCode=0 Nov 02 10:04:01.327546 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.327515 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" Nov 02 10:04:01.327699 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.327615 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" Nov 02 10:04:01.332421 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.332393 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" event={"ID":"5b086519-b3f1-4ade-9e5a-7996dd032e34","Type":"ContainerDied","Data":"458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19"} Nov 02 10:04:01.332574 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.332447 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j" event={"ID":"5b086519-b3f1-4ade-9e5a-7996dd032e34","Type":"ContainerDied","Data":"2a0b66c13ede5c41a9484feef250a25c36ac87b0564b9d75b3c6c36746de676d"} Nov 02 10:04:01.332574 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.332472 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx" event={"ID":"30112bc5-2fef-4629-8944-601ad5b1c87b","Type":"ContainerDied","Data":"66b0a4d57f1dfc6fafd5c6e549362eebeccfa72d0a0b81af2dfb26027a72fa9a"} Nov 02 10:04:01.332574 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.332475 4467 scope.go:122] "RemoveContainer" containerID="458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19" Nov 02 10:04:01.347884 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.347865 4467 scope.go:122] "RemoveContainer" containerID="458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19" Nov 02 10:04:01.348225 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:01.348204 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19\": container with ID starting with 458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19 not found: ID does not exist" containerID="458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19" Nov 02 10:04:01.348283 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.348236 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19"} err="failed to get container status \"458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19\": rpc error: code = NotFound desc = could not find container \"458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19\": container with ID starting with 458256f3ab9a0b1f6ba4cc00ae62dccfe4521bc1e927f23b3a83b56914dbad19 not found: ID does not exist" Nov 02 10:04:01.348283 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.348255 4467 scope.go:122] "RemoveContainer" containerID="0fcf10ae18ee87efdef751a3d8454097b01f24bc94f7e6f408953093592afcc0" Nov 02 10:04:01.358510 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.358488 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx"] Nov 02 10:04:01.364100 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.364069 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6f8dxhwfx"] Nov 02 10:04:01.378420 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.378390 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j"] Nov 02 10:04:01.380313 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.380292 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-64cck995j"] Nov 02 10:04:01.552554 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:01.552474 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:02.240161 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:02.240128 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:03.051787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:03.051754 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:03.330815 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:03.330779 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30112bc5-2fef-4629-8944-601ad5b1c87b" path="/var/lib/kubelet/pods/30112bc5-2fef-4629-8944-601ad5b1c87b/volumes" Nov 02 10:04:03.331252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:03.331237 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b086519-b3f1-4ade-9e5a-7996dd032e34" path="/var/lib/kubelet/pods/5b086519-b3f1-4ade-9e5a-7996dd032e34/volumes" Nov 02 10:04:03.748865 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:03.748781 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:04.549808 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:04.549772 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:05.259314 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.259281 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:05.283115 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.283065 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:04:05.478463 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.478195 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:05.497663 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.497628 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq"] Nov 02 10:04:05.498692 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.498654 4467 cpu_manager.go:414] "containerMap: removing container" podUID="69aa7fba-fb2f-45bd-a3ae-ef602853efdd" containerName="discovery" Nov 02 10:04:05.498922 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.498876 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69aa7fba-fb2f-45bd-a3ae-ef602853efdd" containerName="discovery" Nov 02 10:04:05.499048 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.499036 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5b086519-b3f1-4ade-9e5a-7996dd032e34" containerName="payload-processing" Nov 02 10:04:05.499126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.499116 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b086519-b3f1-4ade-9e5a-7996dd032e34" containerName="payload-processing" Nov 02 10:04:05.499208 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.499199 4467 cpu_manager.go:414] "containerMap: removing container" podUID="73965bc2-1f43-4a38-8eca-69919bda0d5f" containerName="payload-processing" Nov 02 10:04:05.499281 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.499272 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73965bc2-1f43-4a38-8eca-69919bda0d5f" containerName="payload-processing" Nov 02 10:04:05.499366 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.499355 4467 cpu_manager.go:414] "containerMap: removing container" podUID="30112bc5-2fef-4629-8944-601ad5b1c87b" containerName="payload-processing" Nov 02 10:04:05.499449 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.499438 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30112bc5-2fef-4629-8944-601ad5b1c87b" containerName="payload-processing" Nov 02 10:04:05.506043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.506006 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="69aa7fba-fb2f-45bd-a3ae-ef602853efdd" containerName="discovery" Nov 02 10:04:05.506285 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.506266 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b086519-b3f1-4ade-9e5a-7996dd032e34" containerName="payload-processing" Nov 02 10:04:05.506396 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.506381 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="73965bc2-1f43-4a38-8eca-69919bda0d5f" containerName="payload-processing" Nov 02 10:04:05.506492 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.506481 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="30112bc5-2fef-4629-8944-601ad5b1c87b" containerName="payload-processing" Nov 02 10:04:05.529555 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.529384 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq"] Nov 02 10:04:05.529555 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.529510 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.538797 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.537973 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\"" Nov 02 10:04:05.538797 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.538078 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\"" Nov 02 10:04:05.538797 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.538122 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\"" Nov 02 10:04:05.538797 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.538191 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-zbbxs\"" Nov 02 10:04:05.538797 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.538224 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\"" Nov 02 10:04:05.538797 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.538262 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-8eb9fb76a028\"" Nov 02 10:04:05.625528 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.625494 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjb2\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-kube-api-access-jdjb2\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.626045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.625543 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.626045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.625647 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.626045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.625705 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.662035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.661989 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:05.745910 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.727314 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.745910 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.727525 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjb2\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-kube-api-access-jdjb2\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.745910 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.727572 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.745910 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.728919 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.745910 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.729448 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.745910 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.732482 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.749969 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.747444 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.764798 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.764721 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjb2\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-kube-api-access-jdjb2\") pod \"payload-processing-external-model-e2e-extmod-a-629129-74879g4jq\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.848657 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.848615 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:05.899973 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.899530 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg"] Nov 02 10:04:05.904963 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.904913 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:05.929774 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:05.929735 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg"] Nov 02 10:04:06.036305 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.036226 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s24z\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-kube-api-access-9s24z\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.036567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.036383 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.036567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.036456 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.036567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.036498 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.138255 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.138149 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s24z\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-kube-api-access-9s24z\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.138453 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.138299 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.138453 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.138367 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.138453 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.138408 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.139530 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.139449 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.139668 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.139535 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.142107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.142076 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.151671 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.151642 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s24z\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-kube-api-access-9s24z\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.199421 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.199388 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq"] Nov 02 10:04:06.203656 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:04:06.203611 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10f3cd1_bf53_4df7_9241_24b96f46423d.slice/crio-e88b0678f66436ed398e1126cd8246b6c658d4ae785a7463f74232441a5abe98 WatchSource:0}: Error finding container e88b0678f66436ed398e1126cd8246b6c658d4ae785a7463f74232441a5abe98: Status 404 returned error can't find the container with id e88b0678f66436ed398e1126cd8246b6c658d4ae785a7463f74232441a5abe98 Nov 02 10:04:06.234869 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.234843 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:06.375498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.375457 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" event={"ID":"e10f3cd1-bf53-4df7-9241-24b96f46423d","Type":"ContainerStarted","Data":"8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28"} Nov 02 10:04:06.376053 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.375588 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" event={"ID":"e10f3cd1-bf53-4df7-9241-24b96f46423d","Type":"ContainerStarted","Data":"e88b0678f66436ed398e1126cd8246b6c658d4ae785a7463f74232441a5abe98"} Nov 02 10:04:06.376178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.376156 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:06.403250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.403113 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" podStartSLOduration=1.4030875 podStartE2EDuration="1.4030875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:04:06.396965703 +0000 UTC m=+1725.236907713" watchObservedRunningTime="2026-11-02 10:04:06.4030875 +0000 UTC m=+1725.243029495" Nov 02 10:04:06.440192 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.440157 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:06.524156 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:06.524132 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg"] Nov 02 10:04:06.527583 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:04:06.527554 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode741e669_4f7c_4cfe_8b22_4ffefe734a7d.slice/crio-7c0745cbefd91fc7f915d823e646bec333f81ac6f43398b3dfb6936af258b3e8 WatchSource:0}: Error finding container 7c0745cbefd91fc7f915d823e646bec333f81ac6f43398b3dfb6936af258b3e8: Status 404 returned error can't find the container with id 7c0745cbefd91fc7f915d823e646bec333f81ac6f43398b3dfb6936af258b3e8 Nov 02 10:04:07.151082 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:07.151044 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:07.382459 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:07.382420 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" event={"ID":"e741e669-4f7c-4cfe-8b22-4ffefe734a7d","Type":"ContainerStarted","Data":"4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d"} Nov 02 10:04:07.382672 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:07.382485 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" event={"ID":"e741e669-4f7c-4cfe-8b22-4ffefe734a7d","Type":"ContainerStarted","Data":"7c0745cbefd91fc7f915d823e646bec333f81ac6f43398b3dfb6936af258b3e8"} Nov 02 10:04:07.382771 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:07.382746 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:07.404363 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:07.404255 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" podStartSLOduration=2.404243324 podStartE2EDuration="2.404243324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:04:07.40186642 +0000 UTC m=+1726.241808416" watchObservedRunningTime="2026-11-02 10:04:07.404243324 +0000 UTC m=+1726.244185318" Nov 02 10:04:07.835945 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:07.835905 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:08.714721 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:08.714682 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:09.480646 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:09.480603 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:10.149991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:10.149953 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:10.186621 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:10.186536 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:04:10.847073 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:10.847024 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:11.557077 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:11.557041 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:11.904540 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:11.904450 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:12.365787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:12.365755 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:13.067015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:13.066983 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:13.868454 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:13.868423 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:14.553957 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:14.553885 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:15.283840 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:15.283804 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:15.966069 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:15.966031 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:16.770993 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:16.770950 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:17.386611 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:17.386585 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:17.576376 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:17.576337 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:18.167625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:18.167592 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:04:18.362613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:18.362575 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:18.436361 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:18.436274 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:18.558975 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:18.546652 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p"] Nov 02 10:04:19.283536 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:19.283508 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:20.067619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:20.067581 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:20.860268 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:20.860234 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:21.572900 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:21.572859 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:22.243284 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:22.243250 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:22.955581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:22.955545 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:23.560013 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.559962 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" podUID="cd8f0279-4d45-4771-8b75-9b469c39d1d6" containerName="payload-processing" containerID="cri-o://8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f" gracePeriod=25 Nov 02 10:04:23.644113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.644075 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:23.832787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.832759 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:04:23.918163 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.918100 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-provider-credentials\") pod \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " Nov 02 10:04:23.918361 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.918301 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-plugins-config-volume\") pod \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " Nov 02 10:04:23.918435 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.918381 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-kube-api-access-qmwj5\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-kube-api-access-qmwj5\") pod \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " Nov 02 10:04:23.918435 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.918429 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-routing-overlay-volume\") pod \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\" (UID: \"cd8f0279-4d45-4771-8b75-9b469c39d1d6\") " Nov 02 10:04:23.918733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.918704 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-plugins-config-volume" pod "cd8f0279-4d45-4771-8b75-9b469c39d1d6" (UID: "cd8f0279-4d45-4771-8b75-9b469c39d1d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:23.918853 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.918766 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-routing-overlay-volume" pod "cd8f0279-4d45-4771-8b75-9b469c39d1d6" (UID: "cd8f0279-4d45-4771-8b75-9b469c39d1d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:23.918853 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.918828 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:23.920363 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.920337 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-provider-credentials" pod "cd8f0279-4d45-4771-8b75-9b469c39d1d6" (UID: "cd8f0279-4d45-4771-8b75-9b469c39d1d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:23.920463 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:23.920401 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-kube-api-access-qmwj5" pod "cd8f0279-4d45-4771-8b75-9b469c39d1d6" (UID: "cd8f0279-4d45-4771-8b75-9b469c39d1d6"). InnerVolumeSpecName "kube-api-access-qmwj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:24.020384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.020348 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwj5\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-kube-api-access-qmwj5\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:24.020384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.020378 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f0279-4d45-4771-8b75-9b469c39d1d6-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:24.020384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.020387 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd8f0279-4d45-4771-8b75-9b469c39d1d6-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:24.382966 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.382930 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:24.600711 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.600680 4467 generic.go:356] "Generic (PLEG): container finished" podID="cd8f0279-4d45-4771-8b75-9b469c39d1d6" containerID="8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f" exitCode=0 Nov 02 10:04:24.600711 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.600719 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" event={"ID":"cd8f0279-4d45-4771-8b75-9b469c39d1d6","Type":"ContainerDied","Data":"8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f"} Nov 02 10:04:24.600985 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.600769 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" event={"ID":"cd8f0279-4d45-4771-8b75-9b469c39d1d6","Type":"ContainerDied","Data":"6b6189cd0744893073fa5ec560edab09614c8bb50a56b988219e448ae8668e74"} Nov 02 10:04:24.600985 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.600793 4467 scope.go:122] "RemoveContainer" containerID="8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f" Nov 02 10:04:24.600985 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.600795 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p" Nov 02 10:04:24.610349 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.610329 4467 scope.go:122] "RemoveContainer" containerID="8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f" Nov 02 10:04:24.610625 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:24.610605 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f\": container with ID starting with 8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f not found: ID does not exist" containerID="8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f" Nov 02 10:04:24.610677 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.610632 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f"} err="failed to get container status \"8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f\": rpc error: code = NotFound desc = could not find container \"8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f\": container with ID starting with 8e50621548ecee9c1c67277f373fc48d0fd1171dfff4eb73916ba7e82602f98f not found: ID does not exist" Nov 02 10:04:24.625513 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.625482 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p"] Nov 02 10:04:24.629511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:24.629486 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb5g7m8p"] Nov 02 10:04:25.047080 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:25.047047 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:25.331581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:25.331550 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8f0279-4d45-4771-8b75-9b469c39d1d6" path="/var/lib/kubelet/pods/cd8f0279-4d45-4771-8b75-9b469c39d1d6/volumes" Nov 02 10:04:25.743088 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:25.743001 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:26.443572 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:26.443540 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:26.856494 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:26.856461 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:04:27.250761 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:27.250657 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:27.933986 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:27.933948 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq"] Nov 02 10:04:27.967628 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:27.967595 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:28.150424 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:28.150385 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:28.251501 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:28.251415 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg"] Nov 02 10:04:28.394825 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:28.394793 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:28.771455 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:28.771391 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:28.955670 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:28.955639 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:29.147341 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:29.147301 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:04:29.159835 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:29.159802 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:29.539534 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:29.539443 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:30.270197 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:30.270166 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:31.141108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:31.141078 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:31.942485 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:31.942452 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:32.642849 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:32.642815 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:32.934463 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:32.934376 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" podUID="e10f3cd1-bf53-4df7-9241-24b96f46423d" containerName="payload-processing" containerID="cri-o://8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28" gracePeriod=25 Nov 02 10:04:33.188198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.188130 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:33.226971 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.226934 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-provider-credentials\") pod \"e10f3cd1-bf53-4df7-9241-24b96f46423d\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " Nov 02 10:04:33.227173 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.227013 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-routing-overlay-volume\") pod \"e10f3cd1-bf53-4df7-9241-24b96f46423d\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " Nov 02 10:04:33.227173 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.227093 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-kube-api-access-jdjb2\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-kube-api-access-jdjb2\") pod \"e10f3cd1-bf53-4df7-9241-24b96f46423d\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " Nov 02 10:04:33.227173 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.227162 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-plugins-config-volume\") pod \"e10f3cd1-bf53-4df7-9241-24b96f46423d\" (UID: \"e10f3cd1-bf53-4df7-9241-24b96f46423d\") " Nov 02 10:04:33.227477 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.227449 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-routing-overlay-volume" pod "e10f3cd1-bf53-4df7-9241-24b96f46423d" (UID: "e10f3cd1-bf53-4df7-9241-24b96f46423d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:33.227836 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.227813 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:33.227980 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.227834 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-plugins-config-volume" pod "e10f3cd1-bf53-4df7-9241-24b96f46423d" (UID: "e10f3cd1-bf53-4df7-9241-24b96f46423d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:33.230067 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.230041 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-provider-credentials" pod "e10f3cd1-bf53-4df7-9241-24b96f46423d" (UID: "e10f3cd1-bf53-4df7-9241-24b96f46423d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:33.230149 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.230078 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-kube-api-access-jdjb2" pod "e10f3cd1-bf53-4df7-9241-24b96f46423d" (UID: "e10f3cd1-bf53-4df7-9241-24b96f46423d"). InnerVolumeSpecName "kube-api-access-jdjb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:33.252351 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.252317 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" podUID="e741e669-4f7c-4cfe-8b22-4ffefe734a7d" containerName="payload-processing" containerID="cri-o://4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d" gracePeriod=25 Nov 02 10:04:33.328691 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.328662 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:33.328691 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.328691 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdjb2\" (UniqueName: \"kubernetes.io/projected/e10f3cd1-bf53-4df7-9241-24b96f46423d-kube-api-access-jdjb2\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:33.328982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.328705 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10f3cd1-bf53-4df7-9241-24b96f46423d-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:33.343527 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.343467 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:33.497580 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.497550 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:33.531117 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.531072 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-provider-credentials\") pod \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " Nov 02 10:04:33.531305 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.531275 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-routing-overlay-volume\") pod \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " Nov 02 10:04:33.531396 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.531333 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-kube-api-access-9s24z\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-kube-api-access-9s24z\") pod \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " Nov 02 10:04:33.531396 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.531386 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-plugins-config-volume\") pod \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\" (UID: \"e741e669-4f7c-4cfe-8b22-4ffefe734a7d\") " Nov 02 10:04:33.531698 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.531672 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-routing-overlay-volume" pod "e741e669-4f7c-4cfe-8b22-4ffefe734a7d" (UID: "e741e669-4f7c-4cfe-8b22-4ffefe734a7d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:33.531982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.531958 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:33.532151 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.532079 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-plugins-config-volume" pod "e741e669-4f7c-4cfe-8b22-4ffefe734a7d" (UID: "e741e669-4f7c-4cfe-8b22-4ffefe734a7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:33.534198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.534173 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-kube-api-access-9s24z" pod "e741e669-4f7c-4cfe-8b22-4ffefe734a7d" (UID: "e741e669-4f7c-4cfe-8b22-4ffefe734a7d"). InnerVolumeSpecName "kube-api-access-9s24z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:33.534198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.534186 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-provider-credentials" pod "e741e669-4f7c-4cfe-8b22-4ffefe734a7d" (UID: "e741e669-4f7c-4cfe-8b22-4ffefe734a7d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:33.632951 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.632911 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s24z\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-kube-api-access-9s24z\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:33.632951 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.632939 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:33.632951 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.632949 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e741e669-4f7c-4cfe-8b22-4ffefe734a7d-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:33.665516 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.665485 4467 generic.go:356] "Generic (PLEG): container finished" podID="e741e669-4f7c-4cfe-8b22-4ffefe734a7d" containerID="4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d" exitCode=0 Nov 02 10:04:33.665719 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.665547 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" Nov 02 10:04:33.665719 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.665573 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" event={"ID":"e741e669-4f7c-4cfe-8b22-4ffefe734a7d","Type":"ContainerDied","Data":"4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d"} Nov 02 10:04:33.665719 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.665622 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg" event={"ID":"e741e669-4f7c-4cfe-8b22-4ffefe734a7d","Type":"ContainerDied","Data":"7c0745cbefd91fc7f915d823e646bec333f81ac6f43398b3dfb6936af258b3e8"} Nov 02 10:04:33.665719 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.665647 4467 scope.go:122] "RemoveContainer" containerID="4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d" Nov 02 10:04:33.666956 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.666930 4467 generic.go:356] "Generic (PLEG): container finished" podID="e10f3cd1-bf53-4df7-9241-24b96f46423d" containerID="8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28" exitCode=0 Nov 02 10:04:33.667065 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.667030 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" event={"ID":"e10f3cd1-bf53-4df7-9241-24b96f46423d","Type":"ContainerDied","Data":"8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28"} Nov 02 10:04:33.667065 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.667060 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" event={"ID":"e10f3cd1-bf53-4df7-9241-24b96f46423d","Type":"ContainerDied","Data":"e88b0678f66436ed398e1126cd8246b6c658d4ae785a7463f74232441a5abe98"} Nov 02 10:04:33.667172 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.667118 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq" Nov 02 10:04:33.674800 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.674780 4467 scope.go:122] "RemoveContainer" containerID="4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d" Nov 02 10:04:33.675179 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:33.675150 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d\": container with ID starting with 4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d not found: ID does not exist" containerID="4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d" Nov 02 10:04:33.675370 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.675189 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d"} err="failed to get container status \"4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d\": rpc error: code = NotFound desc = could not find container \"4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d\": container with ID starting with 4479aad8f9ae2792d8178e68fa409e6803c495d04433b5a0747b6678491ffa7d not found: ID does not exist" Nov 02 10:04:33.675370 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.675237 4467 scope.go:122] "RemoveContainer" containerID="8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28" Nov 02 10:04:33.686459 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.686438 4467 scope.go:122] "RemoveContainer" containerID="8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28" Nov 02 10:04:33.686737 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:33.686717 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28\": container with ID starting with 8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28 not found: ID does not exist" containerID="8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28" Nov 02 10:04:33.686806 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.686748 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28"} err="failed to get container status \"8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28\": rpc error: code = NotFound desc = could not find container \"8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28\": container with ID starting with 8fe20b999ef7fb870f32b2be622de7c38f7436dbe7634798ff194804b42f1f28 not found: ID does not exist" Nov 02 10:04:33.692025 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.692003 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq"] Nov 02 10:04:33.697401 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.697378 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-74879g4jq"] Nov 02 10:04:33.711362 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.711339 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg"] Nov 02 10:04:33.715689 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:33.715669 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmspqg"] Nov 02 10:04:34.047387 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:34.047355 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:34.744852 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:34.744820 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:35.331209 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:35.331175 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10f3cd1-bf53-4df7-9241-24b96f46423d" path="/var/lib/kubelet/pods/e10f3cd1-bf53-4df7-9241-24b96f46423d/volumes" Nov 02 10:04:35.331572 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:35.331559 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e741e669-4f7c-4cfe-8b22-4ffefe734a7d" path="/var/lib/kubelet/pods/e741e669-4f7c-4cfe-8b22-4ffefe734a7d/volumes" Nov 02 10:04:35.445695 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:35.445663 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:35.624157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:35.624071 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:35.943812 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:35.943717 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:36.260693 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:36.260605 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:36.958924 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:36.958828 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:37.799736 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:37.799707 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:38.448522 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:38.448490 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:39.231448 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:39.231401 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:39.854565 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:39.854532 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:40.750287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:40.750252 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:41.544846 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:41.544813 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:42.353630 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:42.353597 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:42.905790 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:42.905761 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:43.093794 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:43.093763 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:43.324076 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:43.324045 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:43.684955 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:43.684854 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:43.977338 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:43.977257 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:44.378804 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:44.378759 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:45.063794 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:45.063754 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:45.755813 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:45.755781 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:45.787315 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:45.787280 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:04:46.563231 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:46.563198 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:47.267413 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:47.267382 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:47.956537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:47.956501 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:48.751455 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:48.751412 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:49.479853 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:49.479814 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:50.017027 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:50.016989 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:50.913194 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:50.913163 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb"] Nov 02 10:04:50.913707 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:50.913419 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" containerID="cri-o://11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248" gracePeriod=30 Nov 02 10:04:50.917230 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:50.916278 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk"] Nov 02 10:04:50.917230 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:50.916445 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" containerID="cri-o://73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20" gracePeriod=30 Nov 02 10:04:51.475551 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:51.475514 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:51.819314 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:51.819271 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:52.075541 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:52.075464 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:52.759291 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:52.759257 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:53.458549 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:53.458514 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:54.264735 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:54.264699 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:54.404934 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:54.404882 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 10:04:54.405108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:54.404958 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 10:04:54.963125 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:54.963088 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:55.675645 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:55.675613 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:56.038632 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.038596 4467 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 10:04:56.039071 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.038679 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 10:04:56.172652 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.172624 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 10:04:56.203049 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.203022 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:04:56.274080 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274045 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-kube-api-access-ckjpr\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-kube-api-access-ckjpr\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274088 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea47c669-9830-478b-8aa5-0e140e5660f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea47c669-9830-478b-8aa5-0e140e5660f6-istiod-ca-cert\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274110 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-socket\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274157 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-envoy\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274186 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-certs\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274241 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-data\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274551 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274311 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-credential-socket\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274551 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274350 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-podinfo\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274551 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274391 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-token\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-token\") pod \"ea47c669-9830-478b-8aa5-0e140e5660f6\" (UID: \"ea47c669-9830-478b-8aa5-0e140e5660f6\") " Nov 02 10:04:56.274551 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274480 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-socket" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.274779 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274713 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-data" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.274779 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274729 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-credential-socket" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.275024 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274934 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.275024 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274962 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.275024 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.274965 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-certs" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.275024 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.275002 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.275319 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.275152 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea47c669-9830-478b-8aa5-0e140e5660f6-istiod-ca-cert" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:56.276666 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.276635 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-podinfo" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:04:56.276756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.276706 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-token" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:56.276846 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.276831 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-kube-api-access-ckjpr" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "kube-api-access-ckjpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:56.277204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.277188 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-envoy" pod "ea47c669-9830-478b-8aa5-0e140e5660f6" (UID: "ea47c669-9830-478b-8aa5-0e140e5660f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.376158 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376125 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-socket\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376339 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376197 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-token\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-token\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376339 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376216 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-podinfo\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376339 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376255 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-envoy\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376339 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376278 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-certs\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376339 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376323 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-kube-api-access-wfxbn\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-kube-api-access-wfxbn\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376389 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9199956c-4e75-43c6-8005-402b8d0c2d1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9199956c-4e75-43c6-8005-402b8d0c2d1f-istiod-ca-cert\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376430 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-credential-socket\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376424 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-socket" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.376754 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376632 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-data\") pod \"9199956c-4e75-43c6-8005-402b8d0c2d1f\" (UID: \"9199956c-4e75-43c6-8005-402b8d0c2d1f\") " Nov 02 10:04:56.376932 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376632 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-certs" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.376932 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376785 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-credential-socket" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.377126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376856 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-data" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.377126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.376851 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9199956c-4e75-43c6-8005-402b8d0c2d1f-istiod-ca-cert" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:56.377126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377049 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377069 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea47c669-9830-478b-8aa5-0e140e5660f6-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377083 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9199956c-4e75-43c6-8005-402b8d0c2d1f-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377097 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377112 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377126 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377143 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377162 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377177 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377189 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckjpr\" (UniqueName: \"kubernetes.io/projected/ea47c669-9830-478b-8aa5-0e140e5660f6-kube-api-access-ckjpr\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.377578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.377204 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea47c669-9830-478b-8aa5-0e140e5660f6-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.378547 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.378526 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-podinfo" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:04:56.378644 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.378546 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-token" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:56.378735 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.378722 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-envoy" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:04:56.378771 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.378736 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-kube-api-access-wfxbn" pod "9199956c-4e75-43c6-8005-402b8d0c2d1f" (UID: "9199956c-4e75-43c6-8005-402b8d0c2d1f"). InnerVolumeSpecName "kube-api-access-wfxbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:56.474145 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.474060 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:56.478317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.478280 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.478317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.478317 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfxbn\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-kube-api-access-wfxbn\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.478532 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.478334 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.478532 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.478349 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9199956c-4e75-43c6-8005-402b8d0c2d1f-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:56.796039 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.795952 4467 generic.go:356] "Generic (PLEG): container finished" podID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerID="73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20" exitCode=0 Nov 02 10:04:56.796039 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.795998 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" Nov 02 10:04:56.796282 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.796033 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" event={"ID":"9199956c-4e75-43c6-8005-402b8d0c2d1f","Type":"ContainerDied","Data":"73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20"} Nov 02 10:04:56.796282 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.796099 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk" event={"ID":"9199956c-4e75-43c6-8005-402b8d0c2d1f","Type":"ContainerDied","Data":"94b2224fa05c16b9ebb8b6bc2b025891c0082e278c128802fc12076ddbc5cde5"} Nov 02 10:04:56.796282 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.796116 4467 scope.go:122] "RemoveContainer" containerID="73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20" Nov 02 10:04:56.797991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.797968 4467 generic.go:356] "Generic (PLEG): container finished" podID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerID="11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248" exitCode=0 Nov 02 10:04:56.798112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.798008 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" Nov 02 10:04:56.798112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.798018 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" event={"ID":"ea47c669-9830-478b-8aa5-0e140e5660f6","Type":"ContainerDied","Data":"11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248"} Nov 02 10:04:56.798112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.798057 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb" event={"ID":"ea47c669-9830-478b-8aa5-0e140e5660f6","Type":"ContainerDied","Data":"8a1d484d10756af89c443b2618a3323f5872b7a660f5a35a56b59a602e99f8d4"} Nov 02 10:04:56.806954 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.806929 4467 scope.go:122] "RemoveContainer" containerID="73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20" Nov 02 10:04:56.807237 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:56.807215 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20\": container with ID starting with 73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20 not found: ID does not exist" containerID="73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20" Nov 02 10:04:56.807323 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.807255 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20"} err="failed to get container status \"73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20\": rpc error: code = NotFound desc = could not find container \"73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20\": container with ID starting with 73367ed8d994d4fe0d8b52eb4bca3af6cf3ef408ab157c1f07c3090f76a90d20 not found: ID does not exist" Nov 02 10:04:56.807323 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.807279 4467 scope.go:122] "RemoveContainer" containerID="11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248" Nov 02 10:04:56.815720 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.815703 4467 scope.go:122] "RemoveContainer" containerID="5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063" Nov 02 10:04:56.826621 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.826591 4467 scope.go:122] "RemoveContainer" containerID="11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248" Nov 02 10:04:56.826979 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.826726 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb"] Nov 02 10:04:56.828625 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:56.828603 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248\": container with ID starting with 11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248 not found: ID does not exist" containerID="11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248" Nov 02 10:04:56.828723 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.828632 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248"} err="failed to get container status \"11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248\": rpc error: code = NotFound desc = could not find container \"11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248\": container with ID starting with 11f12591de21870ba8748154e86f347e9a800de12905ca30dcd656e62cc44248 not found: ID does not exist" Nov 02 10:04:56.828723 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.828650 4467 scope.go:122] "RemoveContainer" containerID="5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063" Nov 02 10:04:56.828961 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:56.828940 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063\": container with ID starting with 5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063 not found: ID does not exist" containerID="5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063" Nov 02 10:04:56.829029 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.828972 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063"} err="failed to get container status \"5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063\": rpc error: code = NotFound desc = could not find container \"5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063\": container with ID starting with 5b7aa61cd97c856e1981ccbe3712db51cf6db546c1a1070261294485201f6063 not found: ID does not exist" Nov 02 10:04:56.829865 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.829844 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-q7ntb"] Nov 02 10:04:56.840915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.840876 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk"] Nov 02 10:04:56.845644 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:56.845618 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-vrphk"] Nov 02 10:04:56.857051 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:56.857017 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:04:57.293652 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:57.293620 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:57.331495 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:57.331463 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" path="/var/lib/kubelet/pods/9199956c-4e75-43c6-8005-402b8d0c2d1f/volumes" Nov 02 10:04:57.331949 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:57.331934 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" path="/var/lib/kubelet/pods/ea47c669-9830-478b-8aa5-0e140e5660f6/volumes" Nov 02 10:04:57.705526 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:57.705495 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:57.749918 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:57.749865 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8"] Nov 02 10:04:57.750335 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:57.750100 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" podUID="75a15cbe-3d8e-4f96-abbc-e9f91a37596c" containerName="maas-api" containerID="cri-o://71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd" gracePeriod=30 Nov 02 10:04:57.890392 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:57.890359 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:58.035631 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.035604 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 10:04:58.198758 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.198727 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-maas-api-tls\") pod \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " Nov 02 10:04:58.199008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.198831 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-metrics-certs\") pod \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " Nov 02 10:04:58.199008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.198886 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-openshift-service-ca\") pod \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " Nov 02 10:04:58.199008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.198949 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-kube-api-access-xb9d5\" (UniqueName: \"kubernetes.io/projected/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-kube-api-access-xb9d5\") pod \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\" (UID: \"75a15cbe-3d8e-4f96-abbc-e9f91a37596c\") " Nov 02 10:04:58.199335 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.199309 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-openshift-service-ca" pod "75a15cbe-3d8e-4f96-abbc-e9f91a37596c" (UID: "75a15cbe-3d8e-4f96-abbc-e9f91a37596c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:04:58.201093 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.201063 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-maas-api-tls" pod "75a15cbe-3d8e-4f96-abbc-e9f91a37596c" (UID: "75a15cbe-3d8e-4f96-abbc-e9f91a37596c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:04:58.201093 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.201077 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-metrics-certs" pod "75a15cbe-3d8e-4f96-abbc-e9f91a37596c" (UID: "75a15cbe-3d8e-4f96-abbc-e9f91a37596c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:04:58.201216 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.201182 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-kube-api-access-xb9d5" pod "75a15cbe-3d8e-4f96-abbc-e9f91a37596c" (UID: "75a15cbe-3d8e-4f96-abbc-e9f91a37596c"). InnerVolumeSpecName "kube-api-access-xb9d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:04:58.300482 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.300380 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:58.300482 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.300421 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:58.300482 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.300436 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:58.300482 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.300452 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb9d5\" (UniqueName: \"kubernetes.io/projected/75a15cbe-3d8e-4f96-abbc-e9f91a37596c-kube-api-access-xb9d5\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:04:58.814109 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.814073 4467 generic.go:356] "Generic (PLEG): container finished" podID="75a15cbe-3d8e-4f96-abbc-e9f91a37596c" containerID="71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd" exitCode=0 Nov 02 10:04:58.814296 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.814120 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" Nov 02 10:04:58.814296 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.814121 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" event={"ID":"75a15cbe-3d8e-4f96-abbc-e9f91a37596c","Type":"ContainerDied","Data":"71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd"} Nov 02 10:04:58.814296 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.814176 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8" event={"ID":"75a15cbe-3d8e-4f96-abbc-e9f91a37596c","Type":"ContainerDied","Data":"352cd49035754a6f162624f2feea159cd6110167d39e6a93c5d8ce9ed7884822"} Nov 02 10:04:58.814296 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.814206 4467 scope.go:122] "RemoveContainer" containerID="71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd" Nov 02 10:04:58.824037 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.824015 4467 scope.go:122] "RemoveContainer" containerID="71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd" Nov 02 10:04:58.824336 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:04:58.824314 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd\": container with ID starting with 71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd not found: ID does not exist" containerID="71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd" Nov 02 10:04:58.824409 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.824348 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd"} err="failed to get container status \"71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd\": rpc error: code = NotFound desc = could not find container \"71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd\": container with ID starting with 71831d7c8bb4fd1ed2177d1ed78e5ae52369a82698bb4d41c684e8efef2ce8dd not found: ID does not exist" Nov 02 10:04:58.839198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.839172 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8"] Nov 02 10:04:58.841352 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:58.841332 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-204fba-c989d8c6-pq5l8"] Nov 02 10:04:59.271871 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:59.271771 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:04:59.330995 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:59.330949 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a15cbe-3d8e-4f96-abbc-e9f91a37596c" path="/var/lib/kubelet/pods/75a15cbe-3d8e-4f96-abbc-e9f91a37596c/volumes" Nov 02 10:04:59.790884 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:04:59.790846 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:05:02.696529 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:02.696494 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:05.008065 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.008012 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:05.184974 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.184939 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:05.445482 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.445449 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:05.686836 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.685560 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq"] Nov 02 10:05:05.686836 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.685826 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" containerID="cri-o://cfbd4c7b2b382eee245272815afc9c3eb937cc28e3fcfe97f73b81be7e4186bd" gracePeriod=30 Nov 02 10:05:05.688587 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.688551 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr"] Nov 02 10:05:05.690639 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.690603 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" containerID="cri-o://acbc2427dbf193a31f16d19a08bee59637c8a08f59a309e3766768df21ed53f7" gracePeriod=30 Nov 02 10:05:05.691727 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.691703 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b"] Nov 02 10:05:05.692325 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.692295 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" containerID="cri-o://af310a99beea0845348ff632c86ece1b1767d3a60793034d59ef65ea3cddd14b" gracePeriod=30 Nov 02 10:05:05.821379 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:05.821337 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:07.935417 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:07.935384 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5njvh_94af169c-dda1-4cce-acdd-eb7070b2c30e/manager/0.log" Nov 02 10:05:08.166023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:08.165994 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-52pxc_2506bfdc-b907-401c-ad87-d390372a5e08/limitador/0.log" Nov 02 10:05:09.337158 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:09.337129 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f755f6879-xh4h5_30722de2-3a23-49f9-9672-50446962e7e2/manager/0.log" Nov 02 10:05:10.527022 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.526996 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2htfm_eadb4244-9782-409a-b4d2-e875d845c023/server/0.log" Nov 02 10:05:10.624541 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.624504 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 10:05:10.624705 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.624569 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 10:05:10.716982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.716951 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d2kkp_77449e74-83c7-4592-bea4-48a7d20412d8/manager/0.log" Nov 02 10:05:10.886321 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.886287 4467 generic.go:356] "Generic (PLEG): container finished" podID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerID="acbc2427dbf193a31f16d19a08bee59637c8a08f59a309e3766768df21ed53f7" exitCode=0 Nov 02 10:05:10.886471 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.886351 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" event={"ID":"4d2cfa19-5036-404b-8f71-df1bb7c37b45","Type":"ContainerDied","Data":"acbc2427dbf193a31f16d19a08bee59637c8a08f59a309e3766768df21ed53f7"} Nov 02 10:05:10.888225 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.888202 4467 generic.go:356] "Generic (PLEG): container finished" podID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerID="cfbd4c7b2b382eee245272815afc9c3eb937cc28e3fcfe97f73b81be7e4186bd" exitCode=0 Nov 02 10:05:10.888376 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.888282 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" event={"ID":"c3264c12-364d-44ab-a875-d19ae898b5ab","Type":"ContainerDied","Data":"cfbd4c7b2b382eee245272815afc9c3eb937cc28e3fcfe97f73b81be7e4186bd"} Nov 02 10:05:10.890305 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.890288 4467 generic.go:356] "Generic (PLEG): container finished" podID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerID="af310a99beea0845348ff632c86ece1b1767d3a60793034d59ef65ea3cddd14b" exitCode=0 Nov 02 10:05:10.890401 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.890312 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" event={"ID":"04c69cf9-ca53-4857-8b3d-47bf9b3aa331","Type":"ContainerDied","Data":"af310a99beea0845348ff632c86ece1b1767d3a60793034d59ef65ea3cddd14b"} Nov 02 10:05:10.927560 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.927532 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7shdp_925f1379-43e8-4594-ae67-ae2cf3bafab3/proxy/0.log" Nov 02 10:05:10.938907 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.938869 4467 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 10:05:10.939035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.938937 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 10:05:10.960811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:10.960780 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 10:05:11.025608 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.025573 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-kube-api-access-sr252\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-kube-api-access-sr252\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.025811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.025661 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-credential-socket\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.025811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.025753 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-token\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-token\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.025811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.025798 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-socket\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.026023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.025822 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-data\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.026023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.025871 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-podinfo\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.026023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.025929 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-certs\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.026023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.025983 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-envoy\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.026023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026009 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-credential-socket" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.026273 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026014 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3264c12-364d-44ab-a875-d19ae898b5ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3264c12-364d-44ab-a875-d19ae898b5ab-istiod-ca-cert\") pod \"c3264c12-364d-44ab-a875-d19ae898b5ab\" (UID: \"c3264c12-364d-44ab-a875-d19ae898b5ab\") " Nov 02 10:05:11.026273 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026072 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-socket" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.026273 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026160 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-data" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.026413 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026361 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-certs" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.026413 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026378 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3264c12-364d-44ab-a875-d19ae898b5ab-istiod-ca-cert" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:05:11.026553 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026535 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.026622 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026559 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.026622 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026575 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.026622 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026589 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.026622 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.026602 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3264c12-364d-44ab-a875-d19ae898b5ab-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.028667 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.028602 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-envoy" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.028851 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.028785 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-token" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:11.029047 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.029015 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-podinfo" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:05:11.029333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.029311 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-kube-api-access-sr252" pod "c3264c12-364d-44ab-a875-d19ae898b5ab" (UID: "c3264c12-364d-44ab-a875-d19ae898b5ab"). InnerVolumeSpecName "kube-api-access-sr252". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:11.127761 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.127719 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr252\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-kube-api-access-sr252\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.127761 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.127744 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.127761 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.127754 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.127761 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.127763 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3264c12-364d-44ab-a875-d19ae898b5ab-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.361425 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.361401 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 10:05:11.371088 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.371066 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 10:05:11.431357 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431277 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-kube-api-access-q7dzh\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-kube-api-access-q7dzh\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.431357 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431330 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-podinfo\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.431357 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431353 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-token\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-token\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.431609 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431379 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-envoy\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.431609 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431499 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-podinfo\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.431609 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431580 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-kube-api-access-bx7q8\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-kube-api-access-bx7q8\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.431783 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431646 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istiod-ca-cert\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.431783 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431675 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istiod-ca-cert\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.431783 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431707 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-certs\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.431783 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431754 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-credential-socket\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.432032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431790 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-data\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.432032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431847 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-socket\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.432032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431875 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-envoy\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.432032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431945 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-socket\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.432032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.431975 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-certs\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.432032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.432024 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-data\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.432332 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.432061 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-token\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-token\") pod \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\" (UID: \"4d2cfa19-5036-404b-8f71-df1bb7c37b45\") " Nov 02 10:05:11.432332 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.432092 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-credential-socket\") pod \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\" (UID: \"04c69cf9-ca53-4857-8b3d-47bf9b3aa331\") " Nov 02 10:05:11.432332 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.432222 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istiod-ca-cert" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:05:11.432592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.432569 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.432821 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.432797 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-certs" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.432992 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.432964 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istiod-ca-cert" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:05:11.433264 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.433240 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-socket" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.433399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.433254 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-credential-socket" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.433486 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.433435 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-data" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.433613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.433592 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-socket" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.433849 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.433830 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-certs" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.434117 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.434095 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-envoy" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.434210 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.434195 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-data" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.434272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.434244 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-kube-api-access-q7dzh" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "kube-api-access-q7dzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:11.434446 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.434394 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-token" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:11.434446 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.434407 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-credential-socket" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.434537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.434500 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-kube-api-access-bx7q8" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "kube-api-access-bx7q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:11.434684 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.434660 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-podinfo" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:05:11.435486 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.435459 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-podinfo" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:05:11.435652 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.435633 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-envoy" pod "04c69cf9-ca53-4857-8b3d-47bf9b3aa331" (UID: "04c69cf9-ca53-4857-8b3d-47bf9b3aa331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:05:11.436208 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.436185 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-token" pod "4d2cfa19-5036-404b-8f71-df1bb7c37b45" (UID: "4d2cfa19-5036-404b-8f71-df1bb7c37b45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:11.533345 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533307 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533345 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533342 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533357 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx7q8\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-kube-api-access-bx7q8\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533371 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533383 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533395 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533407 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533419 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533432 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533444 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533459 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533473 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533486 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533501 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533517 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7dzh\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-kube-api-access-q7dzh\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533530 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2cfa19-5036-404b-8f71-df1bb7c37b45-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.533843 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.533543 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c69cf9-ca53-4857-8b3d-47bf9b3aa331-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:11.897087 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.897055 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" Nov 02 10:05:11.897087 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.897061 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b" event={"ID":"04c69cf9-ca53-4857-8b3d-47bf9b3aa331","Type":"ContainerDied","Data":"6083801eaaf30cde4f3d7056a5c2ac624bac2af7a09525ea23ba5fc181f818f5"} Nov 02 10:05:11.897343 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.897157 4467 scope.go:122] "RemoveContainer" containerID="af310a99beea0845348ff632c86ece1b1767d3a60793034d59ef65ea3cddd14b" Nov 02 10:05:11.898467 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.898402 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" Nov 02 10:05:11.898467 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.898408 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr" event={"ID":"4d2cfa19-5036-404b-8f71-df1bb7c37b45","Type":"ContainerDied","Data":"033f4333e5330fc47aad1aa0b2cd0291583a0ef75c0fef98dd20f4847f0b6560"} Nov 02 10:05:11.900023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.899996 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" event={"ID":"c3264c12-364d-44ab-a875-d19ae898b5ab","Type":"ContainerDied","Data":"bd6f9c81dbb0239e0cbf20a836c07fcc57e67e1ab1de0dbe60ad47000e808f0c"} Nov 02 10:05:11.900136 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.900066 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq" Nov 02 10:05:11.903711 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.903689 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:11.908032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.908013 4467 scope.go:122] "RemoveContainer" containerID="acbc2427dbf193a31f16d19a08bee59637c8a08f59a309e3766768df21ed53f7" Nov 02 10:05:11.917036 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.917018 4467 scope.go:122] "RemoveContainer" containerID="cfbd4c7b2b382eee245272815afc9c3eb937cc28e3fcfe97f73b81be7e4186bd" Nov 02 10:05:11.923397 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.923376 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq"] Nov 02 10:05:11.930581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.930560 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-2q2rq"] Nov 02 10:05:11.943290 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.943261 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr"] Nov 02 10:05:11.947450 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.947412 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-9q2xr"] Nov 02 10:05:11.959007 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.958981 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b"] Nov 02 10:05:11.965698 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:11.965639 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-75t4b"] Nov 02 10:05:13.331358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:13.331326 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" path="/var/lib/kubelet/pods/04c69cf9-ca53-4857-8b3d-47bf9b3aa331/volumes" Nov 02 10:05:13.331773 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:13.331760 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" path="/var/lib/kubelet/pods/4d2cfa19-5036-404b-8f71-df1bb7c37b45/volumes" Nov 02 10:05:13.332231 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:13.332217 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" path="/var/lib/kubelet/pods/c3264c12-364d-44ab-a875-d19ae898b5ab/volumes" Nov 02 10:05:15.531916 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:15.531869 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:15.953764 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:15.953722 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:16.133867 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:16.133827 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:18.687673 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:18.687640 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:18.872765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:18.872729 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:19.817412 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817381 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn"] Nov 02 10:05:19.817828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817801 4467 cpu_manager.go:414] "containerMap: removing container" podUID="e10f3cd1-bf53-4df7-9241-24b96f46423d" containerName="payload-processing" Nov 02 10:05:19.817828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817814 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10f3cd1-bf53-4df7-9241-24b96f46423d" containerName="payload-processing" Nov 02 10:05:19.817828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817828 4467 cpu_manager.go:414] "containerMap: removing container" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817833 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817843 4467 cpu_manager.go:414] "containerMap: removing container" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817848 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817854 4467 cpu_manager.go:414] "containerMap: removing container" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817860 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817867 4467 cpu_manager.go:414] "containerMap: removing container" podUID="75a15cbe-3d8e-4f96-abbc-e9f91a37596c" containerName="maas-api" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817872 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a15cbe-3d8e-4f96-abbc-e9f91a37596c" containerName="maas-api" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817878 4467 cpu_manager.go:414] "containerMap: removing container" podUID="e741e669-4f7c-4cfe-8b22-4ffefe734a7d" containerName="payload-processing" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817882 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e741e669-4f7c-4cfe-8b22-4ffefe734a7d" containerName="payload-processing" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817927 4467 cpu_manager.go:414] "containerMap: removing container" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817932 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817939 4467 cpu_manager.go:414] "containerMap: removing container" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817944 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817982 4467 cpu_manager.go:414] "containerMap: removing container" podUID="cd8f0279-4d45-4771-8b75-9b469c39d1d6" containerName="payload-processing" Nov 02 10:05:19.818034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.817990 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8f0279-4d45-4771-8b75-9b469c39d1d6" containerName="payload-processing" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.818090 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.818104 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2cfa19-5036-404b-8f71-df1bb7c37b45" containerName="istio-proxy" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.818113 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8f0279-4d45-4771-8b75-9b469c39d1d6" containerName="payload-processing" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.818122 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.819296 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10f3cd1-bf53-4df7-9241-24b96f46423d" containerName="payload-processing" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.819324 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c69cf9-ca53-4857-8b3d-47bf9b3aa331" containerName="istio-proxy" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.819343 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="9199956c-4e75-43c6-8005-402b8d0c2d1f" containerName="istio-proxy" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.819352 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3264c12-364d-44ab-a875-d19ae898b5ab" containerName="istio-proxy" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.819360 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a15cbe-3d8e-4f96-abbc-e9f91a37596c" containerName="maas-api" Nov 02 10:05:19.819841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.819411 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="e741e669-4f7c-4cfe-8b22-4ffefe734a7d" containerName="payload-processing" Nov 02 10:05:19.823181 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.823160 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:19.837518 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.834267 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b9ae0feb-serving-cert\"" Nov 02 10:05:19.837518 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.837126 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn"] Nov 02 10:05:19.905563 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.905500 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm"] Nov 02 10:05:19.905982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.905967 4467 cpu_manager.go:414] "containerMap: removing container" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" Nov 02 10:05:19.905982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.905981 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea47c669-9830-478b-8aa5-0e140e5660f6" containerName="istio-proxy" Nov 02 10:05:19.909598 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.909576 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:19.916329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.916300 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcz2z\" (UniqueName: \"kubernetes.io/projected/77700ab9-9e32-4c67-ad0a-1e149888a0ec-kube-api-access-pcz2z\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:19.916425 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.916360 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-metrics-certs\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:19.916425 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.916399 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-maas-api-tls\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:19.916425 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.916416 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77700ab9-9e32-4c67-ad0a-1e149888a0ec-openshift-service-ca\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:19.917417 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.917393 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b9ae0feb\"" Nov 02 10:05:19.918341 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.917945 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b9ae0feb-dockercfg-7q7bc\"" Nov 02 10:05:19.920592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.920566 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm"] Nov 02 10:05:19.934490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.934465 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:19.950369 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.950339 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf"] Nov 02 10:05:19.954104 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.954083 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:19.964492 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:19.964469 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf"] Nov 02 10:05:20.017851 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.017814 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-maas-api-tls\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.018048 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.017861 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77700ab9-9e32-4c67-ad0a-1e149888a0ec-openshift-service-ca\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.018048 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.017989 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d321f63-67db-4dfc-8383-d2d064065030-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm\" (UID: \"6d321f63-67db-4dfc-8383-d2d064065030\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:20.018048 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.018035 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86671cf9-7174-4998-be4d-facc1b2d6051-plugins-config-volume\") pod \"payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf\" (UID: \"86671cf9-7174-4998-be4d-facc1b2d6051\") " pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:20.018198 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.018068 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8h5\" (UniqueName: \"kubernetes.io/projected/86671cf9-7174-4998-be4d-facc1b2d6051-kube-api-access-nd8h5\") pod \"payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf\" (UID: \"86671cf9-7174-4998-be4d-facc1b2d6051\") " pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:20.018247 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.018225 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcz2z\" (UniqueName: \"kubernetes.io/projected/77700ab9-9e32-4c67-ad0a-1e149888a0ec-kube-api-access-pcz2z\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.018300 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.018264 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnlqg\" (UniqueName: \"kubernetes.io/projected/6d321f63-67db-4dfc-8383-d2d064065030-kube-api-access-pnlqg\") pod \"payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm\" (UID: \"6d321f63-67db-4dfc-8383-d2d064065030\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:20.018350 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.018298 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-metrics-certs\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.018687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.018663 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77700ab9-9e32-4c67-ad0a-1e149888a0ec-openshift-service-ca\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.020941 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.020907 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-metrics-certs\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.021295 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.021252 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-maas-api-tls\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.033058 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.033030 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcz2z\" (UniqueName: \"kubernetes.io/projected/77700ab9-9e32-4c67-ad0a-1e149888a0ec-kube-api-access-pcz2z\") pod \"maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.118999 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.118881 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d321f63-67db-4dfc-8383-d2d064065030-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm\" (UID: \"6d321f63-67db-4dfc-8383-d2d064065030\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:20.118999 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.118972 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86671cf9-7174-4998-be4d-facc1b2d6051-plugins-config-volume\") pod \"payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf\" (UID: \"86671cf9-7174-4998-be4d-facc1b2d6051\") " pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:20.118999 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.118996 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8h5\" (UniqueName: \"kubernetes.io/projected/86671cf9-7174-4998-be4d-facc1b2d6051-kube-api-access-nd8h5\") pod \"payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf\" (UID: \"86671cf9-7174-4998-be4d-facc1b2d6051\") " pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:20.119250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.119055 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnlqg\" (UniqueName: \"kubernetes.io/projected/6d321f63-67db-4dfc-8383-d2d064065030-kube-api-access-pnlqg\") pod \"payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm\" (UID: \"6d321f63-67db-4dfc-8383-d2d064065030\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:20.119611 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.119588 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d321f63-67db-4dfc-8383-d2d064065030-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm\" (UID: \"6d321f63-67db-4dfc-8383-d2d064065030\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:20.119784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.119764 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86671cf9-7174-4998-be4d-facc1b2d6051-plugins-config-volume\") pod \"payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf\" (UID: \"86671cf9-7174-4998-be4d-facc1b2d6051\") " pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:20.124203 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.124178 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:20.131329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.131311 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8h5\" (UniqueName: \"kubernetes.io/projected/86671cf9-7174-4998-be4d-facc1b2d6051-kube-api-access-nd8h5\") pod \"payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf\" (UID: \"86671cf9-7174-4998-be4d-facc1b2d6051\") " pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:20.131485 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.131464 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnlqg\" (UniqueName: \"kubernetes.io/projected/6d321f63-67db-4dfc-8383-d2d064065030-kube-api-access-pnlqg\") pod \"payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm\" (UID: \"6d321f63-67db-4dfc-8383-d2d064065030\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:20.142218 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.142196 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:20.223685 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.223651 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:20.268773 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.268738 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:20.418312 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.418288 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn"] Nov 02 10:05:20.420440 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:05:20.420409 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77700ab9_9e32_4c67_ad0a_1e149888a0ec.slice/crio-877e5b1b71c72240679d17b6f831f9459798b744d267940b8c8877cc852a031d WatchSource:0}: Error finding container 877e5b1b71c72240679d17b6f831f9459798b744d267940b8c8877cc852a031d: Status 404 returned error can't find the container with id 877e5b1b71c72240679d17b6f831f9459798b744d267940b8c8877cc852a031d Nov 02 10:05:20.421769 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.421747 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 10:05:20.499183 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.498811 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm"] Nov 02 10:05:20.501399 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:05:20.501363 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d321f63_67db_4dfc_8383_d2d064065030.slice/crio-52bb4df89fcccb890544a3dca3a97ec72c6e1fb81379d013f69ff637ce150920 WatchSource:0}: Error finding container 52bb4df89fcccb890544a3dca3a97ec72c6e1fb81379d013f69ff637ce150920: Status 404 returned error can't find the container with id 52bb4df89fcccb890544a3dca3a97ec72c6e1fb81379d013f69ff637ce150920 Nov 02 10:05:20.553107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.553008 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf"] Nov 02 10:05:20.556402 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:05:20.556374 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86671cf9_7174_4998_be4d_facc1b2d6051.slice/crio-bf0eb7fb599ad995c100428bff9980762d860b00d006569ef2bbb1b87e9e313e WatchSource:0}: Error finding container bf0eb7fb599ad995c100428bff9980762d860b00d006569ef2bbb1b87e9e313e: Status 404 returned error can't find the container with id bf0eb7fb599ad995c100428bff9980762d860b00d006569ef2bbb1b87e9e313e Nov 02 10:05:20.883728 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.883694 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:20.953144 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.953109 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" event={"ID":"6d321f63-67db-4dfc-8383-d2d064065030","Type":"ContainerStarted","Data":"0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4"} Nov 02 10:05:20.953336 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.953207 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" event={"ID":"6d321f63-67db-4dfc-8383-d2d064065030","Type":"ContainerStarted","Data":"52bb4df89fcccb890544a3dca3a97ec72c6e1fb81379d013f69ff637ce150920"} Nov 02 10:05:20.953336 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.953242 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:20.955385 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.955348 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" event={"ID":"86671cf9-7174-4998-be4d-facc1b2d6051","Type":"ContainerStarted","Data":"0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7"} Nov 02 10:05:20.955515 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.955408 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" event={"ID":"86671cf9-7174-4998-be4d-facc1b2d6051","Type":"ContainerStarted","Data":"bf0eb7fb599ad995c100428bff9980762d860b00d006569ef2bbb1b87e9e313e"} Nov 02 10:05:20.955616 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.955600 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:20.956768 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.956747 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" event={"ID":"77700ab9-9e32-4c67-ad0a-1e149888a0ec","Type":"ContainerStarted","Data":"877e5b1b71c72240679d17b6f831f9459798b744d267940b8c8877cc852a031d"} Nov 02 10:05:20.970713 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.970662 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" podStartSLOduration=1.970646908 podStartE2EDuration="1.970646908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:05:20.970214536 +0000 UTC m=+1799.810156532" watchObservedRunningTime="2026-11-02 10:05:20.970646908 +0000 UTC m=+1799.810588911" Nov 02 10:05:20.990450 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:20.990405 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" podStartSLOduration=1.990392849 podStartE2EDuration="1.990392849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:05:20.988580443 +0000 UTC m=+1799.828522439" watchObservedRunningTime="2026-11-02 10:05:20.990392849 +0000 UTC m=+1799.830334885" Nov 02 10:05:21.654442 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:21.654413 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:21.963671 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:21.963576 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" event={"ID":"77700ab9-9e32-4c67-ad0a-1e149888a0ec","Type":"ContainerStarted","Data":"3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758"} Nov 02 10:05:21.964099 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:21.963968 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:21.983830 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:21.983785 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" podStartSLOduration=2.321840381 podStartE2EDuration="2.983773502s" totalImagesPullingTime="661.933121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:05:21.982537749 +0000 UTC m=+1800.822479745" watchObservedRunningTime="2026-11-02 10:05:21.983773502 +0000 UTC m=+1800.823715497" Nov 02 10:05:22.356244 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:22.356207 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:22.887782 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:22.887738 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:05:23.180541 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:23.180447 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:23.847223 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:23.847186 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:24.645090 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:24.645058 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:25.477373 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:25.477339 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:26.275092 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:26.275057 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:26.854155 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:05:26.854121 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:05:26.955561 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:26.955529 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:27.754623 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:27.754585 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:27.973681 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:27.973654 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:28.497955 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:28.497916 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:29.350279 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:29.350244 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:30.175385 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:30.175351 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:30.852520 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:30.852485 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:31.549340 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:31.549312 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:31.966458 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:31.966426 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:31.966846 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:31.966491 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:32.256479 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:32.256396 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:32.956694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:32.956658 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:33.761515 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:33.761483 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:34.454741 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:34.454694 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:35.300369 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:35.300330 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:36.055338 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:36.055307 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:36.754634 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:36.754601 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:37.455587 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:37.455553 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:38.155141 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:38.155108 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:38.883312 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:38.883275 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:39.748007 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:39.747961 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:40.525179 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:40.525148 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:41.279427 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:41.279396 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:41.945642 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:41.945609 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:42.644936 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:42.644881 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:43.428613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:43.428579 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:44.197880 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:44.197849 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:44.722252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:44.722208 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf"] Nov 02 10:05:44.736739 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:44.736708 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm"] Nov 02 10:05:44.859680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:44.859643 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:45.037362 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.037277 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:45.248371 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.248332 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn"] Nov 02 10:05:45.248778 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.248610 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" podUID="77700ab9-9e32-4c67-ad0a-1e149888a0ec" containerName="maas-api" containerID="cri-o://3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758" gracePeriod=30 Nov 02 10:05:45.396336 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.396273 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:45.514106 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.514080 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:45.561601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.561568 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77700ab9-9e32-4c67-ad0a-1e149888a0ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77700ab9-9e32-4c67-ad0a-1e149888a0ec-openshift-service-ca\") pod \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " Nov 02 10:05:45.561772 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.561609 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77700ab9-9e32-4c67-ad0a-1e149888a0ec-kube-api-access-pcz2z\" (UniqueName: \"kubernetes.io/projected/77700ab9-9e32-4c67-ad0a-1e149888a0ec-kube-api-access-pcz2z\") pod \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " Nov 02 10:05:45.561772 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.561664 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-metrics-certs\") pod \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " Nov 02 10:05:45.561926 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.561793 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-maas-api-tls\") pod \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\" (UID: \"77700ab9-9e32-4c67-ad0a-1e149888a0ec\") " Nov 02 10:05:45.562010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.561975 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77700ab9-9e32-4c67-ad0a-1e149888a0ec-openshift-service-ca" pod "77700ab9-9e32-4c67-ad0a-1e149888a0ec" (UID: "77700ab9-9e32-4c67-ad0a-1e149888a0ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:05:45.562168 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.562146 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77700ab9-9e32-4c67-ad0a-1e149888a0ec-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:45.563959 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.563930 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-maas-api-tls" pod "77700ab9-9e32-4c67-ad0a-1e149888a0ec" (UID: "77700ab9-9e32-4c67-ad0a-1e149888a0ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:05:45.564096 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.563961 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-metrics-certs" pod "77700ab9-9e32-4c67-ad0a-1e149888a0ec" (UID: "77700ab9-9e32-4c67-ad0a-1e149888a0ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:05:45.564329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.564311 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77700ab9-9e32-4c67-ad0a-1e149888a0ec-kube-api-access-pcz2z" pod "77700ab9-9e32-4c67-ad0a-1e149888a0ec" (UID: "77700ab9-9e32-4c67-ad0a-1e149888a0ec"). InnerVolumeSpecName "kube-api-access-pcz2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:45.594187 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.594159 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:45.662713 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.662626 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:45.662713 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.662659 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcz2z\" (UniqueName: \"kubernetes.io/projected/77700ab9-9e32-4c67-ad0a-1e149888a0ec-kube-api-access-pcz2z\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:45.662713 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.662672 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77700ab9-9e32-4c67-ad0a-1e149888a0ec-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:45.775863 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:45.775829 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:46.079821 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.079784 4467 generic.go:356] "Generic (PLEG): container finished" podID="77700ab9-9e32-4c67-ad0a-1e149888a0ec" containerID="3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758" exitCode=0 Nov 02 10:05:46.080035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.079836 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" Nov 02 10:05:46.080035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.079865 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" event={"ID":"77700ab9-9e32-4c67-ad0a-1e149888a0ec","Type":"ContainerDied","Data":"3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758"} Nov 02 10:05:46.080035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.079951 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn" event={"ID":"77700ab9-9e32-4c67-ad0a-1e149888a0ec","Type":"ContainerDied","Data":"877e5b1b71c72240679d17b6f831f9459798b744d267940b8c8877cc852a031d"} Nov 02 10:05:46.080035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.079982 4467 scope.go:122] "RemoveContainer" containerID="3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758" Nov 02 10:05:46.093207 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.093161 4467 scope.go:122] "RemoveContainer" containerID="3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758" Nov 02 10:05:46.093566 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:05:46.093541 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758\": container with ID starting with 3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758 not found: ID does not exist" containerID="3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758" Nov 02 10:05:46.093624 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.093577 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758"} err="failed to get container status \"3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758\": rpc error: code = NotFound desc = could not find container \"3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758\": container with ID starting with 3f2fc05661f989ce2d359ba93106cfda06644c960207c0abf8d622ac3790f758 not found: ID does not exist" Nov 02 10:05:46.108081 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.106202 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn"] Nov 02 10:05:46.108081 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.108035 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9ae0feb-6f5654c5b9-xqwqn"] Nov 02 10:05:46.948396 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:46.948350 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:05:47.331483 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:47.331449 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77700ab9-9e32-4c67-ad0a-1e149888a0ec" path="/var/lib/kubelet/pods/77700ab9-9e32-4c67-ad0a-1e149888a0ec/volumes" Nov 02 10:05:49.723536 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:49.723478 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" podUID="86671cf9-7174-4998-be4d-facc1b2d6051" containerName="payload-processing" containerID="cri-o://0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7" gracePeriod=25 Nov 02 10:05:49.737034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:49.736995 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" podUID="6d321f63-67db-4dfc-8383-d2d064065030" containerName="payload-pre-processing" containerID="cri-o://0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4" gracePeriod=25 Nov 02 10:05:49.991192 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:49.991123 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:50.005547 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.005520 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86671cf9-7174-4998-be4d-facc1b2d6051-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86671cf9-7174-4998-be4d-facc1b2d6051-plugins-config-volume\") pod \"86671cf9-7174-4998-be4d-facc1b2d6051\" (UID: \"86671cf9-7174-4998-be4d-facc1b2d6051\") " Nov 02 10:05:50.005717 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.005650 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86671cf9-7174-4998-be4d-facc1b2d6051-kube-api-access-nd8h5\" (UniqueName: \"kubernetes.io/projected/86671cf9-7174-4998-be4d-facc1b2d6051-kube-api-access-nd8h5\") pod \"86671cf9-7174-4998-be4d-facc1b2d6051\" (UID: \"86671cf9-7174-4998-be4d-facc1b2d6051\") " Nov 02 10:05:50.005989 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.005961 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86671cf9-7174-4998-be4d-facc1b2d6051-plugins-config-volume" pod "86671cf9-7174-4998-be4d-facc1b2d6051" (UID: "86671cf9-7174-4998-be4d-facc1b2d6051"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:05:50.007985 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.007951 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86671cf9-7174-4998-be4d-facc1b2d6051-kube-api-access-nd8h5" pod "86671cf9-7174-4998-be4d-facc1b2d6051" (UID: "86671cf9-7174-4998-be4d-facc1b2d6051"). InnerVolumeSpecName "kube-api-access-nd8h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:50.029230 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.029199 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:50.106951 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.106903 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d321f63-67db-4dfc-8383-d2d064065030-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d321f63-67db-4dfc-8383-d2d064065030-plugins-config-volume\") pod \"6d321f63-67db-4dfc-8383-d2d064065030\" (UID: \"6d321f63-67db-4dfc-8383-d2d064065030\") " Nov 02 10:05:50.107128 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.106990 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d321f63-67db-4dfc-8383-d2d064065030-kube-api-access-pnlqg\" (UniqueName: \"kubernetes.io/projected/6d321f63-67db-4dfc-8383-d2d064065030-kube-api-access-pnlqg\") pod \"6d321f63-67db-4dfc-8383-d2d064065030\" (UID: \"6d321f63-67db-4dfc-8383-d2d064065030\") " Nov 02 10:05:50.107327 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.107293 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d321f63-67db-4dfc-8383-d2d064065030-plugins-config-volume" pod "6d321f63-67db-4dfc-8383-d2d064065030" (UID: "6d321f63-67db-4dfc-8383-d2d064065030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:05:50.107465 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.107367 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86671cf9-7174-4998-be4d-facc1b2d6051-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:50.107465 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.107386 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd8h5\" (UniqueName: \"kubernetes.io/projected/86671cf9-7174-4998-be4d-facc1b2d6051-kube-api-access-nd8h5\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:50.109168 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.109145 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d321f63-67db-4dfc-8383-d2d064065030-kube-api-access-pnlqg" pod "6d321f63-67db-4dfc-8383-d2d064065030" (UID: "6d321f63-67db-4dfc-8383-d2d064065030"). InnerVolumeSpecName "kube-api-access-pnlqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:05:50.110673 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.110645 4467 generic.go:356] "Generic (PLEG): container finished" podID="6d321f63-67db-4dfc-8383-d2d064065030" containerID="0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4" exitCode=0 Nov 02 10:05:50.110762 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.110703 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" Nov 02 10:05:50.110762 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.110718 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" event={"ID":"6d321f63-67db-4dfc-8383-d2d064065030","Type":"ContainerDied","Data":"0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4"} Nov 02 10:05:50.110870 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.110769 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm" event={"ID":"6d321f63-67db-4dfc-8383-d2d064065030","Type":"ContainerDied","Data":"52bb4df89fcccb890544a3dca3a97ec72c6e1fb81379d013f69ff637ce150920"} Nov 02 10:05:50.110870 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.110785 4467 scope.go:122] "RemoveContainer" containerID="0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4" Nov 02 10:05:50.112294 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.112270 4467 generic.go:356] "Generic (PLEG): container finished" podID="86671cf9-7174-4998-be4d-facc1b2d6051" containerID="0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7" exitCode=0 Nov 02 10:05:50.112407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.112342 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" Nov 02 10:05:50.112407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.112385 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" event={"ID":"86671cf9-7174-4998-be4d-facc1b2d6051","Type":"ContainerDied","Data":"0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7"} Nov 02 10:05:50.112527 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.112428 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf" event={"ID":"86671cf9-7174-4998-be4d-facc1b2d6051","Type":"ContainerDied","Data":"bf0eb7fb599ad995c100428bff9980762d860b00d006569ef2bbb1b87e9e313e"} Nov 02 10:05:50.120595 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.120561 4467 scope.go:122] "RemoveContainer" containerID="0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4" Nov 02 10:05:50.120942 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:05:50.120919 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4\": container with ID starting with 0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4 not found: ID does not exist" containerID="0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4" Nov 02 10:05:50.121051 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.120950 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4"} err="failed to get container status \"0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4\": rpc error: code = NotFound desc = could not find container \"0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4\": container with ID starting with 0d77abf0198c19aac7cfadea72e0ebba75d0c057d8cd3eb10d184313c14c9ed4 not found: ID does not exist" Nov 02 10:05:50.121051 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.120966 4467 scope.go:122] "RemoveContainer" containerID="0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7" Nov 02 10:05:50.129340 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.129320 4467 scope.go:122] "RemoveContainer" containerID="0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7" Nov 02 10:05:50.129609 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:05:50.129584 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7\": container with ID starting with 0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7 not found: ID does not exist" containerID="0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7" Nov 02 10:05:50.129673 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.129618 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7"} err="failed to get container status \"0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7\": rpc error: code = NotFound desc = could not find container \"0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7\": container with ID starting with 0d85456f399789c2a0db0421811007ced762a5822978859f63c58f683524e1a7 not found: ID does not exist" Nov 02 10:05:50.139112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.139088 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm"] Nov 02 10:05:50.144775 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.144751 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b9ae0feb-586d87c8c9-6hwhm"] Nov 02 10:05:50.155713 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.155693 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf"] Nov 02 10:05:50.160740 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.160718 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b9ae0feb-fcd47598c-l25nf"] Nov 02 10:05:50.208640 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.208609 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d321f63-67db-4dfc-8383-d2d064065030-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:50.208775 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:50.208644 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnlqg\" (UniqueName: \"kubernetes.io/projected/6d321f63-67db-4dfc-8383-d2d064065030-kube-api-access-pnlqg\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:05:51.269528 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:51.269468 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:05:51.330976 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:51.330946 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d321f63-67db-4dfc-8383-d2d064065030" path="/var/lib/kubelet/pods/6d321f63-67db-4dfc-8383-d2d064065030/volumes" Nov 02 10:05:51.331329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:05:51.331315 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86671cf9-7174-4998-be4d-facc1b2d6051" path="/var/lib/kubelet/pods/86671cf9-7174-4998-be4d-facc1b2d6051/volumes" Nov 02 10:05:56.917382 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:05:56.917346 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:06:02.283115 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:02.283074 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:06:02.496107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:02.496078 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:06:12.153882 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:12.153799 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:06:24.822580 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:24.822534 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:06:26.707802 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.707762 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:06:26.766082 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766051 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b555b5c85-sr2mh"] Nov 02 10:06:26.766523 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766508 4467 cpu_manager.go:414] "containerMap: removing container" podUID="77700ab9-9e32-4c67-ad0a-1e149888a0ec" containerName="maas-api" Nov 02 10:06:26.766567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766526 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77700ab9-9e32-4c67-ad0a-1e149888a0ec" containerName="maas-api" Nov 02 10:06:26.766567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766550 4467 cpu_manager.go:414] "containerMap: removing container" podUID="6d321f63-67db-4dfc-8383-d2d064065030" containerName="payload-pre-processing" Nov 02 10:06:26.766567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766555 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d321f63-67db-4dfc-8383-d2d064065030" containerName="payload-pre-processing" Nov 02 10:06:26.766679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766568 4467 cpu_manager.go:414] "containerMap: removing container" podUID="86671cf9-7174-4998-be4d-facc1b2d6051" containerName="payload-processing" Nov 02 10:06:26.766679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766573 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86671cf9-7174-4998-be4d-facc1b2d6051" containerName="payload-processing" Nov 02 10:06:26.766679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766636 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d321f63-67db-4dfc-8383-d2d064065030" containerName="payload-pre-processing" Nov 02 10:06:26.766679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766645 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="86671cf9-7174-4998-be4d-facc1b2d6051" containerName="payload-processing" Nov 02 10:06:26.766679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.766653 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="77700ab9-9e32-4c67-ad0a-1e149888a0ec" containerName="maas-api" Nov 02 10:06:26.771280 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.771262 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.777818 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.777791 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 10:06:26.779228 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.779205 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 10:06:26.779350 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.779338 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrg5v\"" Nov 02 10:06:26.781845 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.781817 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-sr2mh"] Nov 02 10:06:26.823460 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.823433 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 10:06:26.854334 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:06:26.854299 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:06:26.854743 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.854727 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8787154-b48a-4aba-8c76-d1d9d0e434a2-webhook-cert\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.854787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.854767 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8787154-b48a-4aba-8c76-d1d9d0e434a2-metrics-certs\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.854837 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.854787 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9sn\" (UniqueName: \"kubernetes.io/projected/c8787154-b48a-4aba-8c76-d1d9d0e434a2-kube-api-access-fw9sn\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.956329 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.956291 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8787154-b48a-4aba-8c76-d1d9d0e434a2-webhook-cert\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.956524 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.956342 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8787154-b48a-4aba-8c76-d1d9d0e434a2-metrics-certs\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.956524 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.956453 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9sn\" (UniqueName: \"kubernetes.io/projected/c8787154-b48a-4aba-8c76-d1d9d0e434a2-kube-api-access-fw9sn\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.958789 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.958727 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8787154-b48a-4aba-8c76-d1d9d0e434a2-metrics-certs\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.958789 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.958756 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8787154-b48a-4aba-8c76-d1d9d0e434a2-webhook-cert\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:26.969252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:26.969228 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9sn\" (UniqueName: \"kubernetes.io/projected/c8787154-b48a-4aba-8c76-d1d9d0e434a2-kube-api-access-fw9sn\") pod \"maas-controller-5b555b5c85-sr2mh\" (UID: \"c8787154-b48a-4aba-8c76-d1d9d0e434a2\") " pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:27.086200 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:27.086162 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:27.365053 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:27.365027 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-sr2mh"] Nov 02 10:06:27.367121 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:06:27.367087 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8787154_b48a_4aba_8c76_d1d9d0e434a2.slice/crio-e87676531e757a8cd9f219eacf9abe58dec03cd07b27c507633489a3fd79cbdf WatchSource:0}: Error finding container e87676531e757a8cd9f219eacf9abe58dec03cd07b27c507633489a3fd79cbdf: Status 404 returned error can't find the container with id e87676531e757a8cd9f219eacf9abe58dec03cd07b27c507633489a3fd79cbdf Nov 02 10:06:28.305649 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:28.305608 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-sr2mh" event={"ID":"c8787154-b48a-4aba-8c76-d1d9d0e434a2","Type":"ContainerStarted","Data":"e87676531e757a8cd9f219eacf9abe58dec03cd07b27c507633489a3fd79cbdf"} Nov 02 10:06:29.312042 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:29.312007 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-sr2mh" event={"ID":"c8787154-b48a-4aba-8c76-d1d9d0e434a2","Type":"ContainerStarted","Data":"e12f7e7e2ce7bb93ae01f6dff4c03f57894861aef205bd0d3acbb22431242141"} Nov 02 10:06:29.312472 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:29.312098 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:29.334255 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:29.334201 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5b555b5c85-sr2mh" podStartSLOduration=1.6021255079999999 podStartE2EDuration="3.334187946s" totalImagesPullingTime="1.732062438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:06:29.333035086 +0000 UTC m=+1868.172977081" watchObservedRunningTime="2026-11-02 10:06:29.334187946 +0000 UTC m=+1868.174130009" Nov 02 10:06:31.070689 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:31.070656 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:06:40.319383 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:40.319346 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b555b5c85-sr2mh" Nov 02 10:06:40.910751 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:40.910717 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:06:47.176083 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:47.176049 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:06:50.679078 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:50.679037 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:06:50.679492 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:06:50.679132 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:06:56.855023 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:06:56.854983 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:07:04.030582 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:04.030542 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:07:08.789621 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:08.789501 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:07:12.111906 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:12.111864 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:07:12.359278 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:12.359240 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:07:26.855001 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:07:26.854964 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:07:28.771607 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:28.771566 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:07:33.598046 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:33.597995 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:07:33.607168 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:33.598549 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:07:39.034026 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:39.033982 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:07:39.219016 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:39.218931 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:07:56.855394 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:07:56.855358 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:07:57.051631 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:57.051596 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:07:58.746929 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:58.742874 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:07:58.903918 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:58.902068 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr"] Nov 02 10:07:59.055963 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.045105 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr"] Nov 02 10:07:59.055963 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.045246 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:07:59.093916 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.093528 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4a8104f7-kserve-self-signed-certs\"" Nov 02 10:07:59.146923 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.094336 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4a8104f7-epp-sa-dockercfg-6rrkf\"" Nov 02 10:07:59.146923 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.142457 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:07:59.202919 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.202757 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ds8\" (UniqueName: \"kubernetes.io/projected/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-kube-api-access-x9ds8\") pod \"e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr\" (UID: \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:07:59.202919 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.202859 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-tls-certs\") pod \"e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr\" (UID: \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:07:59.306982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.305413 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ds8\" (UniqueName: \"kubernetes.io/projected/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-kube-api-access-x9ds8\") pod \"e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr\" (UID: \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:07:59.306982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.305531 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-tls-certs\") pod \"e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr\" (UID: \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:07:59.320923 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.319646 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-tls-certs\") pod \"e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr\" (UID: \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:07:59.355107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.355073 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ds8\" (UniqueName: \"kubernetes.io/projected/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-kube-api-access-x9ds8\") pod \"e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr\" (UID: \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:07:59.479442 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.479403 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:07:59.917503 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:07:59.917471 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr"] Nov 02 10:07:59.919115 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:07:59.919089 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c61d550_e5de_48b8_a52a_0a638eb3cf4d.slice/crio-1f15ff063f8f966cc2b0b2468c346918bd5ee5449fcb7b3ebf087c92fd46a33c WatchSource:0}: Error finding container 1f15ff063f8f966cc2b0b2468c346918bd5ee5449fcb7b3ebf087c92fd46a33c: Status 404 returned error can't find the container with id 1f15ff063f8f966cc2b0b2468c346918bd5ee5449fcb7b3ebf087c92fd46a33c Nov 02 10:08:00.710173 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:00.710135 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:08:00.885856 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:00.885812 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" event={"ID":"3c61d550-e5de-48b8-a52a-0a638eb3cf4d","Type":"ContainerStarted","Data":"1f15ff063f8f966cc2b0b2468c346918bd5ee5449fcb7b3ebf087c92fd46a33c"} Nov 02 10:08:01.971392 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:01.971350 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:08:02.901530 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:02.901480 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" event={"ID":"3c61d550-e5de-48b8-a52a-0a638eb3cf4d","Type":"ContainerStarted","Data":"aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba"} Nov 02 10:08:02.902572 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:02.902544 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:08:02.925338 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:02.925272 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" podStartSLOduration=2.744675946 podStartE2EDuration="4.925252661s" totalImagesPullingTime="2.180576715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:08:02.922298523 +0000 UTC m=+1961.762240519" watchObservedRunningTime="2026-11-02 10:08:02.925252661 +0000 UTC m=+1961.765194658" Nov 02 10:08:05.687916 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:05.686991 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:08:05.687916 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:05.687057 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:08:12.167408 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:12.167368 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:08:17.243286 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:17.243252 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:08:26.853395 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:08:26.853355 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:08:34.930225 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:34.930197 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:08:41.194615 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:08:41.194561 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:08:56.854201 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:08:56.854166 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:09:01.177942 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:01.177905 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:09:01.600349 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:01.600315 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr"] Nov 02 10:09:01.694665 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:01.694606 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:01.881205 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:01.881104 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:04.944096 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:04.943994 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:09:05.557392 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:05.557349 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:09:05.678079 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:05.678028 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:09:05.678288 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:05.678114 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:09:12.277701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:12.277664 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:14.931157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:14.931120 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:09:16.623730 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:16.623686 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" containerID="cri-o://aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba" gracePeriod=45 Nov 02 10:09:18.517407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:18.517370 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:19.001815 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:19.001785 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:19.188504 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:19.188471 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:22.286225 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:22.286193 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:22.521177 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:22.521148 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:23.506033 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:23.505997 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:23.686288 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:23.686247 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:24.324298 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:24.324262 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:24.930803 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:24.930761 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:09:24.931277 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:24.930867 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:09:25.120178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:25.120146 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:25.811871 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:25.811838 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:26.069037 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:26.068963 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:09:26.581469 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:26.581433 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:26.854425 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:09:26.854344 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:09:27.315710 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:27.314770 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:28.009364 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:28.009309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:28.716394 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:28.716355 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:29.407956 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:29.407923 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:30.101613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:30.101583 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:30.809828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:30.809798 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:31.559101 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:31.559070 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:32.231570 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:32.231537 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:32.403580 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:32.403544 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:32.828671 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:32.828636 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:33.012460 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:33.012428 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:33.194717 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:33.194637 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:33.807167 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:33.807121 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:34.007857 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:34.007812 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:34.872418 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:34.872382 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:34.881921 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:34.881873 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:09:34.931695 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:34.931653 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:09:35.817742 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:35.817704 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:36.634058 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:36.634030 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:37.330548 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:37.330517 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:38.034786 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:38.034753 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:38.718585 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:38.718552 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:39.409971 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:39.409940 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:40.113846 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:40.113809 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:40.812549 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:40.812518 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:41.512204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:41.512172 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:42.208363 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:42.208329 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:42.906691 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:42.906658 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:43.621516 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:43.621483 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:44.415149 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:44.415118 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:44.931247 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:44.931209 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:09:45.117638 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:45.117606 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:45.318739 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:45.318708 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:45.562316 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:45.562285 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:46.103037 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:46.103000 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:46.806420 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:46.806386 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:46.880228 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:46.880190 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:09:47.047111 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.047027 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-tls-certs\") pod \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\" (UID: \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\") " Nov 02 10:09:47.047272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.047173 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-kube-api-access-x9ds8\" (UniqueName: \"kubernetes.io/projected/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-kube-api-access-x9ds8\") pod \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\" (UID: \"3c61d550-e5de-48b8-a52a-0a638eb3cf4d\") " Nov 02 10:09:47.049402 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.049373 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-tls-certs" pod "3c61d550-e5de-48b8-a52a-0a638eb3cf4d" (UID: "3c61d550-e5de-48b8-a52a-0a638eb3cf4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:09:47.049587 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.049565 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-kube-api-access-x9ds8" pod "3c61d550-e5de-48b8-a52a-0a638eb3cf4d" (UID: "3c61d550-e5de-48b8-a52a-0a638eb3cf4d"). InnerVolumeSpecName "kube-api-access-x9ds8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:09:47.148419 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.148381 4467 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-tls-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:09:47.148419 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.148411 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ds8\" (UniqueName: \"kubernetes.io/projected/3c61d550-e5de-48b8-a52a-0a638eb3cf4d-kube-api-access-x9ds8\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:09:47.553581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.553220 4467 generic.go:356] "Generic (PLEG): container finished" podID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerID="aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba" exitCode=0 Nov 02 10:09:47.553581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.553285 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" event={"ID":"3c61d550-e5de-48b8-a52a-0a638eb3cf4d","Type":"ContainerDied","Data":"aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba"} Nov 02 10:09:47.553581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.553381 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" event={"ID":"3c61d550-e5de-48b8-a52a-0a638eb3cf4d","Type":"ContainerDied","Data":"1f15ff063f8f966cc2b0b2468c346918bd5ee5449fcb7b3ebf087c92fd46a33c"} Nov 02 10:09:47.553581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.553439 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr" Nov 02 10:09:47.555176 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.553455 4467 scope.go:122] "RemoveContainer" containerID="aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba" Nov 02 10:09:47.566427 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.566403 4467 scope.go:122] "RemoveContainer" containerID="aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba" Nov 02 10:09:47.566759 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:09:47.566741 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba\": container with ID starting with aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba not found: ID does not exist" containerID="aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba" Nov 02 10:09:47.566818 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.566769 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba"} err="failed to get container status \"aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba\": rpc error: code = NotFound desc = could not find container \"aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba\": container with ID starting with aeea7b3e4b0e82fd4c123e3caea8d5c90c489a44455f5d06f02171108361e8ba not found: ID does not exist" Nov 02 10:09:47.573585 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.573560 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr"] Nov 02 10:09:47.577640 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.577620 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-router-scheduler-75f8877679-x2qwr"] Nov 02 10:09:47.606687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:47.606607 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:48.413234 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:48.413201 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:49.113575 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:49.113544 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:49.331751 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:49.331718 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" path="/var/lib/kubelet/pods/3c61d550-e5de-48b8-a52a-0a638eb3cf4d/volumes" Nov 02 10:09:49.814485 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:49.814453 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:50.625307 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:50.625271 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:51.396199 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:51.396162 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:52.749898 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:52.749847 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:09:53.249045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:53.249006 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:53.617246 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:53.617215 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:54.418993 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:54.418944 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:55.123266 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:55.123228 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:55.837092 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:55.837060 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:56.521881 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:56.521855 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:56.854951 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:09:56.854911 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:09:57.341338 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:57.341307 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:58.023693 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:58.023656 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:58.799473 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:58.799439 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:09:59.507422 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:09:59.507386 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:00.219999 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:00.219966 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:00.901755 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:00.901714 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:01.617595 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:01.617560 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:02.316287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:02.316258 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:03.019239 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:03.019192 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:03.723845 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:03.723816 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:04.416058 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:04.416023 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:05.221605 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:05.221575 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:06.024714 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:06.024676 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:06.816532 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:06.816500 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:07.523595 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:07.523554 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:08.219740 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:08.219708 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:08.933709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:08.933676 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:09.609213 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:09.609166 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:10.313490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:10.313446 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:11.014057 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:11.014019 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:11.740962 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:11.740927 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:12.258412 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:12.258374 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:12.503524 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:12.503491 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:13.227472 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:13.227435 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:13.923452 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:13.923420 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:14.678196 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:14.678166 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:15.422395 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:15.422363 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:15.676854 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:15.676762 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:16.072420 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:16.072374 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:10:16.123491 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:16.123441 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:16.300866 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:16.300830 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:16.921015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:16.920975 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:17.701091 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:17.701055 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:18.617603 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:18.617568 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:19.329284 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:19.329251 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:20.022580 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:20.022546 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:20.709505 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:20.709461 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:20.885173 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:20.885136 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:21.313971 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:21.313923 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:21.497353 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:21.497308 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:22.452757 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:22.452724 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:23.054976 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:23.054938 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:10:26.853824 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:10:26.853779 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:10:28.379558 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:28.379524 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:28.581676 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:28.581623 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:43.318308 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:43.318268 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:43.976357 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:43.976324 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:44.182384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:44.182322 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:44.451632 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:44.451594 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:48.552564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:48.552524 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:49.846061 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:49.846022 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:50.001773 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:50.001736 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:10:50.036461 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:50.036404 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:51.298019 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:51.297987 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:51.477055 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:10:51.477012 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:10:56.854071 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:10:56.854035 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:11:05.085207 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.085173 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs"] Nov 02 10:11:05.085760 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.085742 4467 cpu_manager.go:414] "containerMap: removing container" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" Nov 02 10:11:05.085809 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.085763 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" Nov 02 10:11:05.085856 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.085847 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c61d550-e5de-48b8-a52a-0a638eb3cf4d" containerName="main" Nov 02 10:11:05.089548 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.089519 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.099589 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.099565 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bd1362d4-kserve-self-signed-certs\"" Nov 02 10:11:05.102642 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.102620 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs"] Nov 02 10:11:05.153451 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.153416 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tls-certs\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.153610 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.153479 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-dshm\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.153610 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.153570 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-home\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.153695 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.153630 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-model-cache\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.153695 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.153650 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht64w\" (UniqueName: \"kubernetes.io/projected/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-kube-api-access-ht64w\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.153695 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.153670 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tmp-dir\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.234648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.234598 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:11:05.254157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254120 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-dshm\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.254157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254163 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-home\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.254418 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254200 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-model-cache\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.254418 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254219 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht64w\" (UniqueName: \"kubernetes.io/projected/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-kube-api-access-ht64w\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.254418 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254240 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tmp-dir\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.254418 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254363 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tls-certs\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.254689 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254666 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-model-cache\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.254751 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254685 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-home\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.254751 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.254732 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tmp-dir\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.256453 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.256429 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-dshm\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.256716 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.256697 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tls-certs\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.266487 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.266464 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht64w\" (UniqueName: \"kubernetes.io/projected/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-kube-api-access-ht64w\") pod \"e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.403241 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.403144 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:05.684872 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.684846 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs"] Nov 02 10:11:05.687196 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:11:05.687165 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc642b55_b9a0_40ef_ba9d_1480d9efbd4a.slice/crio-908a73d649ae1712cc089a34e85d32c8da1267c9afa7cedf2e9e26f040898710 WatchSource:0}: Error finding container 908a73d649ae1712cc089a34e85d32c8da1267c9afa7cedf2e9e26f040898710: Status 404 returned error can't find the container with id 908a73d649ae1712cc089a34e85d32c8da1267c9afa7cedf2e9e26f040898710 Nov 02 10:11:05.938245 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.938160 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" event={"ID":"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a","Type":"ContainerStarted","Data":"15d7350aa58c86925db39580882cad7ddaabc93d969280d1a3dc7fbfa3976c87"} Nov 02 10:11:05.938245 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.938202 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" event={"ID":"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a","Type":"ContainerStarted","Data":"908a73d649ae1712cc089a34e85d32c8da1267c9afa7cedf2e9e26f040898710"} Nov 02 10:11:05.957487 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:05.957439 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" podStartSLOduration=0.95742434 podStartE2EDuration="957.42434ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:11:05.957394182 +0000 UTC m=+2144.797336176" watchObservedRunningTime="2026-11-02 10:11:05.95742434 +0000 UTC m=+2144.797366336" Nov 02 10:11:06.404056 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:06.404018 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:06.791818 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:06.791739 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:11:12.392810 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:12.392774 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:11:15.403567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:15.403534 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:15.416125 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:15.416094 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:16.077235 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:16.077199 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:22.906666 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:22.906619 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:11:26.905915 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:11:26.905179 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:11:33.353317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:33.353282 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:11:34.762039 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:34.762002 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:11:35.107752 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:35.107709 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs"] Nov 02 10:11:35.282282 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:35.282241 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:11:41.235916 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:41.222451 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:11:50.131687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.131625 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" podUID="fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" containerName="main" containerID="cri-o://15d7350aa58c86925db39580882cad7ddaabc93d969280d1a3dc7fbfa3976c87" gracePeriod=45 Nov 02 10:11:50.323949 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.323781 4467 generic.go:356] "Generic (PLEG): container finished" podID="fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" containerID="15d7350aa58c86925db39580882cad7ddaabc93d969280d1a3dc7fbfa3976c87" exitCode=0 Nov 02 10:11:50.323949 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.323852 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" event={"ID":"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a","Type":"ContainerDied","Data":"15d7350aa58c86925db39580882cad7ddaabc93d969280d1a3dc7fbfa3976c87"} Nov 02 10:11:50.398219 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.398148 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:50.453996 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.453962 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tmp-dir\") pod \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " Nov 02 10:11:50.454166 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454056 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-kube-api-access-ht64w\" (UniqueName: \"kubernetes.io/projected/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-kube-api-access-ht64w\") pod \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " Nov 02 10:11:50.454166 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454081 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-model-cache\") pod \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " Nov 02 10:11:50.454166 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454103 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tls-certs\") pod \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " Nov 02 10:11:50.454287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454182 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-dshm\") pod \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " Nov 02 10:11:50.454287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454253 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-home\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-home\") pod \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\" (UID: \"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a\") " Nov 02 10:11:50.454287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454251 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tmp-dir" pod "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" (UID: "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:11:50.454437 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454354 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-model-cache" pod "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" (UID: "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:11:50.454601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454580 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-home" pod "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" (UID: "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:11:50.454672 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454601 4467 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tmp-dir\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:11:50.454672 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.454621 4467 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-model-cache\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:11:50.456220 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.456193 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-dshm" pod "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" (UID: "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:11:50.456295 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.456245 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tls-certs" pod "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" (UID: "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:11:50.456333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.456303 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-kube-api-access-ht64w" pod "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" (UID: "fc642b55-b9a0-40ef-ba9d-1480d9efbd4a"). InnerVolumeSpecName "kube-api-access-ht64w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:11:50.555398 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.555359 4467 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-dshm\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:11:50.555398 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.555390 4467 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-home\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:11:50.555398 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.555404 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht64w\" (UniqueName: \"kubernetes.io/projected/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-kube-api-access-ht64w\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:11:50.555649 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.555417 4467 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a-tls-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:11:50.578965 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:50.578918 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:11:51.378055 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:51.378020 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" Nov 02 10:11:51.378491 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:51.378175 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs" event={"ID":"fc642b55-b9a0-40ef-ba9d-1480d9efbd4a","Type":"ContainerDied","Data":"908a73d649ae1712cc089a34e85d32c8da1267c9afa7cedf2e9e26f040898710"} Nov 02 10:11:51.378491 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:51.378221 4467 scope.go:122] "RemoveContainer" containerID="15d7350aa58c86925db39580882cad7ddaabc93d969280d1a3dc7fbfa3976c87" Nov 02 10:11:51.575128 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:51.575088 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs"] Nov 02 10:11:51.581057 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:51.581026 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bd1362d4-kserve-799bf7946f-7nrqs"] Nov 02 10:11:53.358207 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:53.358167 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" path="/var/lib/kubelet/pods/fc642b55-b9a0-40ef-ba9d-1480d9efbd4a/volumes" Nov 02 10:11:55.261094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:11:55.261027 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:11:56.965753 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:11:56.965127 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:12:09.290063 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:09.289979 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:12:11.020137 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:11.020096 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:11.291252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:11.291173 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:11.731414 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:11.731355 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:11.916284 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:11.916247 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:12.436715 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:12.436681 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:15.641756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:15.641722 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:15.826330 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:15.826297 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:16.784221 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.784183 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6"] Nov 02 10:12:16.784744 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.784654 4467 cpu_manager.go:414] "containerMap: removing container" podUID="fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" containerName="main" Nov 02 10:12:16.784744 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.784668 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" containerName="main" Nov 02 10:12:16.784744 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.784733 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc642b55-b9a0-40ef-ba9d-1480d9efbd4a" containerName="main" Nov 02 10:12:16.788403 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.788378 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.794918 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.794873 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2b5a18-serving-cert\"" Nov 02 10:12:16.803656 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.803626 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6"] Nov 02 10:12:16.817384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.817343 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-metrics-certs\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.817592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.817490 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.817592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.817525 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a920f9-c8b1-4129-84f7-d206d320d833-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.817592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.817565 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmcwf\" (UniqueName: \"kubernetes.io/projected/41a920f9-c8b1-4129-84f7-d206d320d833-kube-api-access-dmcwf\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.860058 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.860023 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt"] Nov 02 10:12:16.863699 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.863674 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:16.872850 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.872819 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2b5a18\"" Nov 02 10:12:16.875877 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.875848 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt"] Nov 02 10:12:16.914882 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.914848 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:16.915850 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.915823 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8"] Nov 02 10:12:16.919088 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.919067 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.919178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.919099 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a920f9-c8b1-4129-84f7-d206d320d833-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.919178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.919127 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmcwf\" (UniqueName: \"kubernetes.io/projected/41a920f9-c8b1-4129-84f7-d206d320d833-kube-api-access-dmcwf\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.919296 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.919175 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-metrics-certs\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.919296 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.919206 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6eba3-bb0a-4441-91d9-a3b524dd7915-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt\" (UID: \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:16.919296 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:12:16.919246 4467 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-serving-cert: secret "maas-api-e2e-ipp-a-2b5a18-serving-cert" not found Nov 02 10:12:16.919296 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.919266 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxmt\" (UniqueName: \"kubernetes.io/projected/8be6eba3-bb0a-4441-91d9-a3b524dd7915-kube-api-access-dzxmt\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt\" (UID: \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:16.919493 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:12:16.919330 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls podName:41a920f9-c8b1-4129-84f7-d206d320d833 nodeName:}" failed. No retries permitted until 2026-11-02 10:12:17.419303443 +0000 UTC m=+2216.259245433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls") pod "maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" (UID: "41a920f9-c8b1-4129-84f7-d206d320d833") : secret "maas-api-e2e-ipp-a-2b5a18-serving-cert" not found Nov 02 10:12:16.919744 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.919728 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a920f9-c8b1-4129-84f7-d206d320d833-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.919862 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.919847 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:16.921768 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.921743 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-metrics-certs\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:16.928518 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.928481 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8"] Nov 02 10:12:16.929316 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.929299 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2b5a18-dockercfg-wnt5t\"" Nov 02 10:12:16.935567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:16.935540 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmcwf\" (UniqueName: \"kubernetes.io/projected/41a920f9-c8b1-4129-84f7-d206d320d833-kube-api-access-dmcwf\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:17.020524 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.020485 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8\" (UID: \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:17.020756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.020627 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6eba3-bb0a-4441-91d9-a3b524dd7915-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt\" (UID: \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:17.020756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.020718 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxmt\" (UniqueName: \"kubernetes.io/projected/8be6eba3-bb0a-4441-91d9-a3b524dd7915-kube-api-access-dzxmt\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt\" (UID: \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:17.020880 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.020763 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdhc\" (UniqueName: \"kubernetes.io/projected/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-kube-api-access-xrdhc\") pod \"payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8\" (UID: \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:17.021332 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.021310 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6eba3-bb0a-4441-91d9-a3b524dd7915-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt\" (UID: \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:17.035445 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.035377 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxmt\" (UniqueName: \"kubernetes.io/projected/8be6eba3-bb0a-4441-91d9-a3b524dd7915-kube-api-access-dzxmt\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt\" (UID: \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:17.115931 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.115884 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:17.122239 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.122204 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8\" (UID: \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:17.122399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.122311 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdhc\" (UniqueName: \"kubernetes.io/projected/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-kube-api-access-xrdhc\") pod \"payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8\" (UID: \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:17.123016 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.122990 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8\" (UID: \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:17.134719 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.134691 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdhc\" (UniqueName: \"kubernetes.io/projected/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-kube-api-access-xrdhc\") pod \"payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8\" (UID: \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:17.178823 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.178784 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:17.255263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.255228 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:17.310517 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.310479 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:17.425164 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.425130 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:17.427802 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.427773 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:17.460016 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.459980 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt"] Nov 02 10:12:17.460834 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:12:17.460802 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be6eba3_bb0a_4441_91d9_a3b524dd7915.slice/crio-19f087165aeaff9fdff7d9fa6e97f6d9e2e8c535f5b2b8992352dcd4c3fa8e0a WatchSource:0}: Error finding container 19f087165aeaff9fdff7d9fa6e97f6d9e2e8c535f5b2b8992352dcd4c3fa8e0a: Status 404 returned error can't find the container with id 19f087165aeaff9fdff7d9fa6e97f6d9e2e8c535f5b2b8992352dcd4c3fa8e0a Nov 02 10:12:17.544969 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.544869 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8"] Nov 02 10:12:17.550290 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:12:17.550258 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333e9fe6_7bf2_45ea_94a0_aaf1f50de1a5.slice/crio-d0bb4a0410bb428c95e1ff5dcf10ef3578176df046cd37c80634c93512d13e07 WatchSource:0}: Error finding container d0bb4a0410bb428c95e1ff5dcf10ef3578176df046cd37c80634c93512d13e07: Status 404 returned error can't find the container with id d0bb4a0410bb428c95e1ff5dcf10ef3578176df046cd37c80634c93512d13e07 Nov 02 10:12:17.565019 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.564916 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" event={"ID":"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5","Type":"ContainerStarted","Data":"d0bb4a0410bb428c95e1ff5dcf10ef3578176df046cd37c80634c93512d13e07"} Nov 02 10:12:17.566836 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.566805 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" event={"ID":"8be6eba3-bb0a-4441-91d9-a3b524dd7915","Type":"ContainerStarted","Data":"9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09"} Nov 02 10:12:17.567026 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.566861 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" event={"ID":"8be6eba3-bb0a-4441-91d9-a3b524dd7915","Type":"ContainerStarted","Data":"19f087165aeaff9fdff7d9fa6e97f6d9e2e8c535f5b2b8992352dcd4c3fa8e0a"} Nov 02 10:12:17.567112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.567025 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:17.590802 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.590730 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" podStartSLOduration=1.590710075 podStartE2EDuration="1.590710075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:17.590189886 +0000 UTC m=+2216.430131881" watchObservedRunningTime="2026-11-02 10:12:17.590710075 +0000 UTC m=+2216.430652073" Nov 02 10:12:17.709159 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.709122 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:17.944044 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.943959 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:17.999126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:17.998905 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6"] Nov 02 10:12:18.002006 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:12:18.001970 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a920f9_c8b1_4129_84f7_d206d320d833.slice/crio-19b783ffbce0576a4a41d7cf0879d4276e98f420dcfa729b40cb4a8a0290c2bf WatchSource:0}: Error finding container 19b783ffbce0576a4a41d7cf0879d4276e98f420dcfa729b40cb4a8a0290c2bf: Status 404 returned error can't find the container with id 19b783ffbce0576a4a41d7cf0879d4276e98f420dcfa729b40cb4a8a0290c2bf Nov 02 10:12:18.003423 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:18.003403 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 10:12:18.577613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:18.577373 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" event={"ID":"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5","Type":"ContainerStarted","Data":"62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a"} Nov 02 10:12:18.577613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:18.577557 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:18.579370 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:18.579345 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" event={"ID":"41a920f9-c8b1-4129-84f7-d206d320d833","Type":"ContainerStarted","Data":"19b783ffbce0576a4a41d7cf0879d4276e98f420dcfa729b40cb4a8a0290c2bf"} Nov 02 10:12:18.829720 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:18.829681 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:19.587222 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:19.587161 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" event={"ID":"41a920f9-c8b1-4129-84f7-d206d320d833","Type":"ContainerStarted","Data":"b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd"} Nov 02 10:12:19.587763 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:19.587651 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:19.611332 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:19.611273 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" podStartSLOduration=3.611258496 podStartE2EDuration="3.611258496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:18.600997719 +0000 UTC m=+2217.440939715" watchObservedRunningTime="2026-11-02 10:12:19.611258496 +0000 UTC m=+2218.451200491" Nov 02 10:12:19.611550 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:19.611425 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" podStartSLOduration=3.126432201 podStartE2EDuration="3.61141986s" totalImagesPullingTime="484.987659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:19.609270571 +0000 UTC m=+2218.449212564" watchObservedRunningTime="2026-11-02 10:12:19.61141986 +0000 UTC m=+2218.451361854" Nov 02 10:12:19.628128 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:19.628093 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:19.890543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:19.890454 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:12:20.116988 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:20.116951 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8"] Nov 02 10:12:20.138363 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:20.138326 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt"] Nov 02 10:12:20.287043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:20.286943 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:20.468594 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:20.468560 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:21.235491 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.235454 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:21.746114 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.746074 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk"] Nov 02 10:12:21.749906 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.749869 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:21.754770 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.754742 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:21.759174 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.759146 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk"] Nov 02 10:12:21.784772 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.784734 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lzc\" (UniqueName: \"kubernetes.io/projected/0ca46343-303c-4d67-a855-667123794849-kube-api-access-26lzc\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk\" (UID: \"0ca46343-303c-4d67-a855-667123794849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:21.784963 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.784844 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca46343-303c-4d67-a855-667123794849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk\" (UID: \"0ca46343-303c-4d67-a855-667123794849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:21.887786 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.887742 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca46343-303c-4d67-a855-667123794849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk\" (UID: \"0ca46343-303c-4d67-a855-667123794849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:21.889254 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.888884 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lzc\" (UniqueName: \"kubernetes.io/projected/0ca46343-303c-4d67-a855-667123794849-kube-api-access-26lzc\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk\" (UID: \"0ca46343-303c-4d67-a855-667123794849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:21.890047 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.890023 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca46343-303c-4d67-a855-667123794849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk\" (UID: \"0ca46343-303c-4d67-a855-667123794849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:21.901807 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.901778 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lzc\" (UniqueName: \"kubernetes.io/projected/0ca46343-303c-4d67-a855-667123794849-kube-api-access-26lzc\") pod \"payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk\" (UID: \"0ca46343-303c-4d67-a855-667123794849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:21.934543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:21.934514 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:22.068002 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:22.067968 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:22.211822 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:22.211794 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:22.344666 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:22.344636 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk"] Nov 02 10:12:22.347001 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:12:22.346958 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca46343_303c_4d67_a855_667123794849.slice/crio-e44e3439311bae4ecf1578b2c693e09824f733f2dd988341613869461a0c4092 WatchSource:0}: Error finding container e44e3439311bae4ecf1578b2c693e09824f733f2dd988341613869461a0c4092: Status 404 returned error can't find the container with id e44e3439311bae4ecf1578b2c693e09824f733f2dd988341613869461a0c4092 Nov 02 10:12:22.607626 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:22.607542 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" event={"ID":"0ca46343-303c-4d67-a855-667123794849","Type":"ContainerStarted","Data":"e44e3439311bae4ecf1578b2c693e09824f733f2dd988341613869461a0c4092"} Nov 02 10:12:22.937625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:22.937518 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:23.917618 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:23.917575 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:24.625128 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:24.625092 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:25.139550 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.139454 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" podUID="8be6eba3-bb0a-4441-91d9-a3b524dd7915" containerName="payload-pre-processing" containerID="cri-o://9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09" gracePeriod=25 Nov 02 10:12:25.412212 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.412184 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:25.512452 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.512414 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:25.529982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.529949 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be6eba3-bb0a-4441-91d9-a3b524dd7915-kube-api-access-dzxmt\" (UniqueName: \"kubernetes.io/projected/8be6eba3-bb0a-4441-91d9-a3b524dd7915-kube-api-access-dzxmt\") pod \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\" (UID: \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\") " Nov 02 10:12:25.530182 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.530002 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be6eba3-bb0a-4441-91d9-a3b524dd7915-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6eba3-bb0a-4441-91d9-a3b524dd7915-plugins-config-volume\") pod \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\" (UID: \"8be6eba3-bb0a-4441-91d9-a3b524dd7915\") " Nov 02 10:12:25.530408 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.530385 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be6eba3-bb0a-4441-91d9-a3b524dd7915-plugins-config-volume" pod "8be6eba3-bb0a-4441-91d9-a3b524dd7915" (UID: "8be6eba3-bb0a-4441-91d9-a3b524dd7915"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:12:25.532157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.532129 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be6eba3-bb0a-4441-91d9-a3b524dd7915-kube-api-access-dzxmt" pod "8be6eba3-bb0a-4441-91d9-a3b524dd7915" (UID: "8be6eba3-bb0a-4441-91d9-a3b524dd7915"). InnerVolumeSpecName "kube-api-access-dzxmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:12:25.594122 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.594074 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" podUID="333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5" containerName="payload-processing" containerID="cri-o://62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a" gracePeriod=25 Nov 02 10:12:25.600209 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.600188 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:12:25.627458 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.627406 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" event={"ID":"0ca46343-303c-4d67-a855-667123794849","Type":"ContainerStarted","Data":"90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8"} Nov 02 10:12:25.627647 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.627530 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:25.629202 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.629162 4467 generic.go:356] "Generic (PLEG): container finished" podID="8be6eba3-bb0a-4441-91d9-a3b524dd7915" containerID="9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09" exitCode=0 Nov 02 10:12:25.629202 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.629202 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" event={"ID":"8be6eba3-bb0a-4441-91d9-a3b524dd7915","Type":"ContainerDied","Data":"9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09"} Nov 02 10:12:25.629544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.629211 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" Nov 02 10:12:25.629544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.629238 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt" event={"ID":"8be6eba3-bb0a-4441-91d9-a3b524dd7915","Type":"ContainerDied","Data":"19f087165aeaff9fdff7d9fa6e97f6d9e2e8c535f5b2b8992352dcd4c3fa8e0a"} Nov 02 10:12:25.629544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.629267 4467 scope.go:122] "RemoveContainer" containerID="9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09" Nov 02 10:12:25.631116 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.631088 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzxmt\" (UniqueName: \"kubernetes.io/projected/8be6eba3-bb0a-4441-91d9-a3b524dd7915-kube-api-access-dzxmt\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:12:25.631249 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.631118 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6eba3-bb0a-4441-91d9-a3b524dd7915-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:12:25.665607 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.665553 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" podStartSLOduration=2.175214002 podStartE2EDuration="4.665536282s" totalImagesPullingTime="2.49032228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:25.647046365 +0000 UTC m=+2224.486988361" watchObservedRunningTime="2026-11-02 10:12:25.665536282 +0000 UTC m=+2224.505478279" Nov 02 10:12:25.666794 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.666769 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt"] Nov 02 10:12:25.670270 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.670244 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-7dbfd65bff-xxckt"] Nov 02 10:12:25.707008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.706977 4467 scope.go:122] "RemoveContainer" containerID="9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09" Nov 02 10:12:25.707359 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:12:25.707336 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09\": container with ID starting with 9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09 not found: ID does not exist" containerID="9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09" Nov 02 10:12:25.707452 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.707371 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09"} err="failed to get container status \"9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09\": rpc error: code = NotFound desc = could not find container \"9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09\": container with ID starting with 9e1a79b85b00b96a774fd07b10fc88fb06e73f0366438c07f7b28f214693cb09 not found: ID does not exist" Nov 02 10:12:25.836010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.835985 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:25.933733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.933642 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-kube-api-access-xrdhc\" (UniqueName: \"kubernetes.io/projected/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-kube-api-access-xrdhc\") pod \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\" (UID: \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\") " Nov 02 10:12:25.933733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.933684 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-plugins-config-volume\") pod \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\" (UID: \"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5\") " Nov 02 10:12:25.934200 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.934173 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-plugins-config-volume" pod "333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5" (UID: "333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:12:25.935931 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:25.935901 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-kube-api-access-xrdhc" pod "333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5" (UID: "333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5"). InnerVolumeSpecName "kube-api-access-xrdhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:12:26.034869 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.034838 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdhc\" (UniqueName: \"kubernetes.io/projected/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-kube-api-access-xrdhc\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:12:26.034869 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.034867 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:12:26.223529 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.223444 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:26.635632 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.635601 4467 generic.go:356] "Generic (PLEG): container finished" podID="333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5" containerID="62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a" exitCode=0 Nov 02 10:12:26.635833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.635650 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" Nov 02 10:12:26.635833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.635686 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" event={"ID":"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5","Type":"ContainerDied","Data":"62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a"} Nov 02 10:12:26.635833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.635737 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8" event={"ID":"333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5","Type":"ContainerDied","Data":"d0bb4a0410bb428c95e1ff5dcf10ef3578176df046cd37c80634c93512d13e07"} Nov 02 10:12:26.635833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.635765 4467 scope.go:122] "RemoveContainer" containerID="62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a" Nov 02 10:12:26.644865 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.644844 4467 scope.go:122] "RemoveContainer" containerID="62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a" Nov 02 10:12:26.645175 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:12:26.645157 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a\": container with ID starting with 62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a not found: ID does not exist" containerID="62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a" Nov 02 10:12:26.645233 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.645185 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a"} err="failed to get container status \"62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a\": rpc error: code = NotFound desc = could not find container \"62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a\": container with ID starting with 62cf9311f253a70ab0b2ab96cd9365fea5d23521b3fb3204e7050c5178944d0a not found: ID does not exist" Nov 02 10:12:26.661964 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.661941 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8"] Nov 02 10:12:26.667483 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:26.667460 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-69d476f7dc-8pfc8"] Nov 02 10:12:26.859990 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:12:26.859957 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:12:27.164596 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:27.164560 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:27.331781 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:27.331747 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5" path="/var/lib/kubelet/pods/333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5/volumes" Nov 02 10:12:27.332176 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:27.332168 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be6eba3-bb0a-4441-91d9-a3b524dd7915" path="/var/lib/kubelet/pods/8be6eba3-bb0a-4441-91d9-a3b524dd7915/volumes" Nov 02 10:12:28.010958 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:28.010914 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:28.917181 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:28.917146 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:29.622658 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:29.622620 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:30.503012 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:30.502977 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:31.031942 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:31.031885 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:31.223773 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:31.223741 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:32.209246 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:32.209215 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:32.991815 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:32.991784 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:33.897813 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:33.897778 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:34.631632 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:34.631599 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:35.503183 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:35.503132 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:36.289985 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:36.289947 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:36.640106 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:36.640073 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:12:37.199331 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:37.199276 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:37.945272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:37.945239 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:38.820881 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:38.820851 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:39.533715 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:39.533679 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:40.522045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:40.522012 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:40.697471 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:40.697428 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:41.220413 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:41.220378 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:42.108840 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:42.108805 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:42.292471 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:42.292430 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:42.738662 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:42.738631 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:42.916021 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:42.915974 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:43.731043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:43.731006 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:44.424917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:44.424860 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:45.423187 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:45.423149 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:45.992433 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:45.992400 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:46.205372 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:46.205334 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:46.384486 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:46.384454 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:47.115917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.111422 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8"] Nov 02 10:12:47.115917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.112626 4467 cpu_manager.go:414] "containerMap: removing container" podUID="333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5" containerName="payload-processing" Nov 02 10:12:47.115917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.112649 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5" containerName="payload-processing" Nov 02 10:12:47.115917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.112771 4467 cpu_manager.go:414] "containerMap: removing container" podUID="8be6eba3-bb0a-4441-91d9-a3b524dd7915" containerName="payload-pre-processing" Nov 02 10:12:47.115917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.112784 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be6eba3-bb0a-4441-91d9-a3b524dd7915" containerName="payload-pre-processing" Nov 02 10:12:47.115917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.113609 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="333e9fe6-7bf2-45ea-94a0-aaf1f50de1a5" containerName="payload-processing" Nov 02 10:12:47.115917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.113640 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be6eba3-bb0a-4441-91d9-a3b524dd7915" containerName="payload-pre-processing" Nov 02 10:12:47.134874 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.134841 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8"] Nov 02 10:12:47.135050 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.134946 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.138700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.138663 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-836bba-serving-cert\"" Nov 02 10:12:47.205476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.205444 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:47.214059 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.214031 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx"] Nov 02 10:12:47.217652 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.217621 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:47.225188 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.225169 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-836bba\"" Nov 02 10:12:47.231977 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.231953 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq"] Nov 02 10:12:47.234364 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.234341 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-836bba-dockercfg-g5hmm\"" Nov 02 10:12:47.237229 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.237207 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx"] Nov 02 10:12:47.237337 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.237300 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:47.246068 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.246042 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdq5\" (UniqueName: \"kubernetes.io/projected/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-kube-api-access-5tdq5\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.246187 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.246099 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.246187 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.246127 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.246307 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.246244 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-metrics-certs\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.249451 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.249425 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq"] Nov 02 10:12:47.347700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.347661 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx\" (UID: \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:47.347884 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.347708 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdq5\" (UniqueName: \"kubernetes.io/projected/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-kube-api-access-5tdq5\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.347884 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.347773 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.347996 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.347940 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.347996 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.347977 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxws4\" (UniqueName: \"kubernetes.io/projected/998685e9-4460-4bc6-8bd0-a198d7229d21-kube-api-access-kxws4\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq\" (UID: \"998685e9-4460-4bc6-8bd0-a198d7229d21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:47.348069 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.348040 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998685e9-4460-4bc6-8bd0-a198d7229d21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq\" (UID: \"998685e9-4460-4bc6-8bd0-a198d7229d21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:47.348148 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.348124 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-metrics-certs\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.348324 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.348292 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57gn9\" (UniqueName: \"kubernetes.io/projected/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-kube-api-access-57gn9\") pod \"payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx\" (UID: \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:47.348585 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.348567 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.350229 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.350208 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.350390 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.350373 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-metrics-certs\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.360381 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.360362 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdq5\" (UniqueName: \"kubernetes.io/projected/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-kube-api-access-5tdq5\") pod \"maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.384982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.384911 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:47.449429 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.449392 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:47.449649 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.449626 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998685e9-4460-4bc6-8bd0-a198d7229d21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq\" (UID: \"998685e9-4460-4bc6-8bd0-a198d7229d21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:47.449710 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.449690 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57gn9\" (UniqueName: \"kubernetes.io/projected/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-kube-api-access-57gn9\") pod \"payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx\" (UID: \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:47.449775 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.449761 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx\" (UID: \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:47.449848 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.449832 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxws4\" (UniqueName: \"kubernetes.io/projected/998685e9-4460-4bc6-8bd0-a198d7229d21-kube-api-access-kxws4\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq\" (UID: \"998685e9-4460-4bc6-8bd0-a198d7229d21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:47.450439 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.450413 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998685e9-4460-4bc6-8bd0-a198d7229d21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq\" (UID: \"998685e9-4460-4bc6-8bd0-a198d7229d21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:47.450603 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.450582 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx\" (UID: \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:47.462729 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.462696 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxws4\" (UniqueName: \"kubernetes.io/projected/998685e9-4460-4bc6-8bd0-a198d7229d21-kube-api-access-kxws4\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq\" (UID: \"998685e9-4460-4bc6-8bd0-a198d7229d21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:47.462824 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.462797 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57gn9\" (UniqueName: \"kubernetes.io/projected/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-kube-api-access-57gn9\") pod \"payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx\" (UID: \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:47.533214 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.532590 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:47.561381 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.561346 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:47.733407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.733385 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8"] Nov 02 10:12:47.735441 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:12:47.735411 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5060c7c4_bdf2_46d1_92e8_a1b3ecd96d70.slice/crio-f7aedddf18a37682cbe3074ba992a765923c1b225d2947e3375e7b9cda7da84a WatchSource:0}: Error finding container f7aedddf18a37682cbe3074ba992a765923c1b225d2947e3375e7b9cda7da84a: Status 404 returned error can't find the container with id f7aedddf18a37682cbe3074ba992a765923c1b225d2947e3375e7b9cda7da84a Nov 02 10:12:47.746464 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.746435 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" event={"ID":"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70","Type":"ContainerStarted","Data":"f7aedddf18a37682cbe3074ba992a765923c1b225d2947e3375e7b9cda7da84a"} Nov 02 10:12:47.824543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.824513 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx"] Nov 02 10:12:47.826000 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:12:47.825970 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05817456_2b7f_4e38_8e2d_2ef0aa88f1db.slice/crio-aa562861c7ea37cb570821c0041fe55f48bcccebc9f00dd6eb294cb8bf96fc16 WatchSource:0}: Error finding container aa562861c7ea37cb570821c0041fe55f48bcccebc9f00dd6eb294cb8bf96fc16: Status 404 returned error can't find the container with id aa562861c7ea37cb570821c0041fe55f48bcccebc9f00dd6eb294cb8bf96fc16 Nov 02 10:12:47.844582 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:47.844559 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq"] Nov 02 10:12:47.848700 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:12:47.848675 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998685e9_4460_4bc6_8bd0_a198d7229d21.slice/crio-9a75cdbfbafa5f84e5a5e8bdb875f274955bd1b0f1786f1b298c3633691f35da WatchSource:0}: Error finding container 9a75cdbfbafa5f84e5a5e8bdb875f274955bd1b0f1786f1b298c3633691f35da: Status 404 returned error can't find the container with id 9a75cdbfbafa5f84e5a5e8bdb875f274955bd1b0f1786f1b298c3633691f35da Nov 02 10:12:48.120973 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.120938 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:48.754502 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.754467 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" event={"ID":"05817456-2b7f-4e38-8e2d-2ef0aa88f1db","Type":"ContainerStarted","Data":"d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80"} Nov 02 10:12:48.754711 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.754528 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:48.754711 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.754553 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" event={"ID":"05817456-2b7f-4e38-8e2d-2ef0aa88f1db","Type":"ContainerStarted","Data":"aa562861c7ea37cb570821c0041fe55f48bcccebc9f00dd6eb294cb8bf96fc16"} Nov 02 10:12:48.756410 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.756376 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" event={"ID":"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70","Type":"ContainerStarted","Data":"936a202176da997f4490765f491d0210d1907d7823937b4b2808de4c84bf9034"} Nov 02 10:12:48.756556 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.756437 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:48.758199 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.758173 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" event={"ID":"998685e9-4460-4bc6-8bd0-a198d7229d21","Type":"ContainerStarted","Data":"2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be"} Nov 02 10:12:48.758323 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.758221 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" event={"ID":"998685e9-4460-4bc6-8bd0-a198d7229d21","Type":"ContainerStarted","Data":"9a75cdbfbafa5f84e5a5e8bdb875f274955bd1b0f1786f1b298c3633691f35da"} Nov 02 10:12:48.758323 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.758254 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:48.773142 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.773097 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" podStartSLOduration=1.7730860659999998 podStartE2EDuration="1.773086066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:48.772666445 +0000 UTC m=+2247.612608443" watchObservedRunningTime="2026-11-02 10:12:48.773086066 +0000 UTC m=+2247.613028062" Nov 02 10:12:48.794698 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.794634 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" podStartSLOduration=1.794614245 podStartE2EDuration="1.794614245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:48.790922894 +0000 UTC m=+2247.630864890" watchObservedRunningTime="2026-11-02 10:12:48.794614245 +0000 UTC m=+2247.634556241" Nov 02 10:12:48.816692 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.816642 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" podStartSLOduration=1.359940296 podStartE2EDuration="1.81662572s" totalImagesPullingTime="456.685424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:48.81532098 +0000 UTC m=+2247.655262970" watchObservedRunningTime="2026-11-02 10:12:48.81662572 +0000 UTC m=+2247.656567715" Nov 02 10:12:48.835747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:48.835711 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:49.712019 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:49.711983 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:49.864041 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:49.864009 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:12:50.338344 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:50.338309 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx"] Nov 02 10:12:50.356596 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:50.356559 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq"] Nov 02 10:12:50.498967 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:50.498928 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:50.679238 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:50.679153 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:51.309648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:51.309598 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:52.228811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.228777 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:52.744057 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.744023 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:52.744744 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.744722 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4"] Nov 02 10:12:52.748821 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.748799 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:52.759438 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.759412 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4"] Nov 02 10:12:52.810604 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.810569 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mrh\" (UniqueName: \"kubernetes.io/projected/879f2fd5-92ac-473f-8b10-f66c49ec445a-kube-api-access-c4mrh\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4\" (UID: \"879f2fd5-92ac-473f-8b10-f66c49ec445a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:52.810771 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.810703 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f2fd5-92ac-473f-8b10-f66c49ec445a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4\" (UID: \"879f2fd5-92ac-473f-8b10-f66c49ec445a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:52.911518 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.911481 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f2fd5-92ac-473f-8b10-f66c49ec445a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4\" (UID: \"879f2fd5-92ac-473f-8b10-f66c49ec445a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:52.911680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.911581 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mrh\" (UniqueName: \"kubernetes.io/projected/879f2fd5-92ac-473f-8b10-f66c49ec445a-kube-api-access-c4mrh\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4\" (UID: \"879f2fd5-92ac-473f-8b10-f66c49ec445a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:52.912260 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.912239 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f2fd5-92ac-473f-8b10-f66c49ec445a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4\" (UID: \"879f2fd5-92ac-473f-8b10-f66c49ec445a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:52.923147 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.923110 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:52.924437 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:52.924415 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mrh\" (UniqueName: \"kubernetes.io/projected/879f2fd5-92ac-473f-8b10-f66c49ec445a-kube-api-access-c4mrh\") pod \"payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4\" (UID: \"879f2fd5-92ac-473f-8b10-f66c49ec445a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:53.064025 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:53.063993 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:53.227437 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:53.227403 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:53.340902 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:53.340867 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4"] Nov 02 10:12:53.343276 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:12:53.343243 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879f2fd5_92ac_473f_8b10_f66c49ec445a.slice/crio-327a55eba0a505a4038c468a8476c84f8862e5eb9ca746fbd9ee2e4dc1723cfa WatchSource:0}: Error finding container 327a55eba0a505a4038c468a8476c84f8862e5eb9ca746fbd9ee2e4dc1723cfa: Status 404 returned error can't find the container with id 327a55eba0a505a4038c468a8476c84f8862e5eb9ca746fbd9ee2e4dc1723cfa Nov 02 10:12:53.789081 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:53.788987 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" event={"ID":"879f2fd5-92ac-473f-8b10-f66c49ec445a","Type":"ContainerStarted","Data":"7e69ace79c599e746f95f118b3ed20217bcefdd6fcc2f467a7e5c8764f568ec9"} Nov 02 10:12:53.789081 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:53.789037 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" event={"ID":"879f2fd5-92ac-473f-8b10-f66c49ec445a","Type":"ContainerStarted","Data":"327a55eba0a505a4038c468a8476c84f8862e5eb9ca746fbd9ee2e4dc1723cfa"} Nov 02 10:12:53.789081 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:53.789068 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:12:53.811608 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:53.811064 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" podStartSLOduration=1.811045654 podStartE2EDuration="1.811045654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:53.810926431 +0000 UTC m=+2252.650868439" watchObservedRunningTime="2026-11-02 10:12:53.811045654 +0000 UTC m=+2252.650987651" Nov 02 10:12:54.008793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:54.008758 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:54.768777 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:54.768748 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:12:55.114529 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:55.114483 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:55.768910 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:55.768848 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" podUID="05817456-2b7f-4e38-8e2d-2ef0aa88f1db" containerName="payload-processing" containerID="cri-o://d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80" gracePeriod=25 Nov 02 10:12:55.768910 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:55.768868 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" podUID="998685e9-4460-4bc6-8bd0-a198d7229d21" containerName="payload-pre-processing" containerID="cri-o://2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be" gracePeriod=25 Nov 02 10:12:56.048694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.048667 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:56.067608 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.067579 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:56.081808 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.081788 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:56.143766 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.143735 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998685e9-4460-4bc6-8bd0-a198d7229d21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998685e9-4460-4bc6-8bd0-a198d7229d21-plugins-config-volume\") pod \"998685e9-4460-4bc6-8bd0-a198d7229d21\" (UID: \"998685e9-4460-4bc6-8bd0-a198d7229d21\") " Nov 02 10:12:56.144210 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.143845 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-plugins-config-volume\") pod \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\" (UID: \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\") " Nov 02 10:12:56.144210 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.143879 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998685e9-4460-4bc6-8bd0-a198d7229d21-kube-api-access-kxws4\" (UniqueName: \"kubernetes.io/projected/998685e9-4460-4bc6-8bd0-a198d7229d21-kube-api-access-kxws4\") pod \"998685e9-4460-4bc6-8bd0-a198d7229d21\" (UID: \"998685e9-4460-4bc6-8bd0-a198d7229d21\") " Nov 02 10:12:56.144210 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.144026 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-kube-api-access-57gn9\" (UniqueName: \"kubernetes.io/projected/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-kube-api-access-57gn9\") pod \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\" (UID: \"05817456-2b7f-4e38-8e2d-2ef0aa88f1db\") " Nov 02 10:12:56.144385 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.144219 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998685e9-4460-4bc6-8bd0-a198d7229d21-plugins-config-volume" pod "998685e9-4460-4bc6-8bd0-a198d7229d21" (UID: "998685e9-4460-4bc6-8bd0-a198d7229d21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:12:56.144385 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.144266 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-plugins-config-volume" pod "05817456-2b7f-4e38-8e2d-2ef0aa88f1db" (UID: "05817456-2b7f-4e38-8e2d-2ef0aa88f1db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:12:56.144493 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.144471 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998685e9-4460-4bc6-8bd0-a198d7229d21-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:12:56.144542 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.144499 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:12:56.146103 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.146083 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998685e9-4460-4bc6-8bd0-a198d7229d21-kube-api-access-kxws4" pod "998685e9-4460-4bc6-8bd0-a198d7229d21" (UID: "998685e9-4460-4bc6-8bd0-a198d7229d21"). InnerVolumeSpecName "kube-api-access-kxws4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:12:56.146154 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.146141 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-kube-api-access-57gn9" pod "05817456-2b7f-4e38-8e2d-2ef0aa88f1db" (UID: "05817456-2b7f-4e38-8e2d-2ef0aa88f1db"). InnerVolumeSpecName "kube-api-access-57gn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:12:56.246045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.246010 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxws4\" (UniqueName: \"kubernetes.io/projected/998685e9-4460-4bc6-8bd0-a198d7229d21-kube-api-access-kxws4\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:12:56.246045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.246039 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57gn9\" (UniqueName: \"kubernetes.io/projected/05817456-2b7f-4e38-8e2d-2ef0aa88f1db-kube-api-access-57gn9\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:12:56.807814 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.807779 4467 generic.go:356] "Generic (PLEG): container finished" podID="05817456-2b7f-4e38-8e2d-2ef0aa88f1db" containerID="d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80" exitCode=0 Nov 02 10:12:56.808008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.807833 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" Nov 02 10:12:56.808008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.807869 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" event={"ID":"05817456-2b7f-4e38-8e2d-2ef0aa88f1db","Type":"ContainerDied","Data":"d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80"} Nov 02 10:12:56.808008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.807951 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx" event={"ID":"05817456-2b7f-4e38-8e2d-2ef0aa88f1db","Type":"ContainerDied","Data":"aa562861c7ea37cb570821c0041fe55f48bcccebc9f00dd6eb294cb8bf96fc16"} Nov 02 10:12:56.808008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.807975 4467 scope.go:122] "RemoveContainer" containerID="d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80" Nov 02 10:12:56.810017 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.809745 4467 generic.go:356] "Generic (PLEG): container finished" podID="998685e9-4460-4bc6-8bd0-a198d7229d21" containerID="2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be" exitCode=0 Nov 02 10:12:56.810017 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.809781 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" Nov 02 10:12:56.810017 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.809815 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" event={"ID":"998685e9-4460-4bc6-8bd0-a198d7229d21","Type":"ContainerDied","Data":"2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be"} Nov 02 10:12:56.810017 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.809865 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq" event={"ID":"998685e9-4460-4bc6-8bd0-a198d7229d21","Type":"ContainerDied","Data":"9a75cdbfbafa5f84e5a5e8bdb875f274955bd1b0f1786f1b298c3633691f35da"} Nov 02 10:12:56.819163 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.819142 4467 scope.go:122] "RemoveContainer" containerID="d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80" Nov 02 10:12:56.819508 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:12:56.819470 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80\": container with ID starting with d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80 not found: ID does not exist" containerID="d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80" Nov 02 10:12:56.819630 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.819506 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80"} err="failed to get container status \"d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80\": rpc error: code = NotFound desc = could not find container \"d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80\": container with ID starting with d16eb5439453272c894642d4c549de3e1721798bbf1ce8099029673e42a14e80 not found: ID does not exist" Nov 02 10:12:56.819630 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.819528 4467 scope.go:122] "RemoveContainer" containerID="2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be" Nov 02 10:12:56.828224 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.828204 4467 scope.go:122] "RemoveContainer" containerID="2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be" Nov 02 10:12:56.828472 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:12:56.828451 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be\": container with ID starting with 2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be not found: ID does not exist" containerID="2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be" Nov 02 10:12:56.828578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.828477 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be"} err="failed to get container status \"2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be\": rpc error: code = NotFound desc = could not find container \"2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be\": container with ID starting with 2c4e2a51f926cdb1c671c5da128ef2111f73d01070f30d16f26245a404f367be not found: ID does not exist" Nov 02 10:12:56.837046 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.837025 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx"] Nov 02 10:12:56.842023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.842000 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-5577fcf647-9wmfx"] Nov 02 10:12:56.854155 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.854019 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq"] Nov 02 10:12:56.856025 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:12:56.855996 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:12:56.858413 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.858394 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6777bf66d7-vrfxq"] Nov 02 10:12:56.927904 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:56.927871 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:57.331604 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:57.331566 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05817456-2b7f-4e38-8e2d-2ef0aa88f1db" path="/var/lib/kubelet/pods/05817456-2b7f-4e38-8e2d-2ef0aa88f1db/volumes" Nov 02 10:12:57.332039 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:57.332025 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998685e9-4460-4bc6-8bd0-a198d7229d21" path="/var/lib/kubelet/pods/998685e9-4460-4bc6-8bd0-a198d7229d21/volumes" Nov 02 10:12:57.820712 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:57.820684 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:58.701076 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:58.701036 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:12:59.430717 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:12:59.430680 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:00.304938 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:00.304884 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:01.277585 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:01.277552 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:01.458316 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:01.458280 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:02.009771 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:02.009728 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:02.915620 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:02.915589 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:03.725141 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:03.725107 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:04.433183 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:04.433154 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:04.797272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:04.797197 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:13:05.313995 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:05.313967 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:06.197050 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:06.197015 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:06.915513 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:06.915479 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:07.819922 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:07.819873 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:08.705874 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:08.705840 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:09.428406 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:09.428370 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:10.320574 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:10.320543 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:11.303677 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:11.303644 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:12.006607 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:12.006571 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:12.471297 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:12.471264 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:12.770541 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:12.770463 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-sbd6p_fbe0b6eb-e661-4ac4-ac42-ac13e163f5de/payload-processing/0.log" Nov 02 10:13:12.970095 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:12.970069 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-sbd6p_fbe0b6eb-e661-4ac4-ac42-ac13e163f5de/payload-processing/0.log" Nov 02 10:13:13.050558 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:13.050481 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:13.945025 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:13.944988 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:14.816030 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:14.815991 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:15.352490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:15.352456 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:15.830466 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:15.830435 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:16.820479 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:16.820433 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:17.270513 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:17.270426 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:13:17.705650 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:17.705610 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:18.528768 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:18.528735 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:19.415566 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:19.415529 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:20.159182 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:20.159143 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:21.029332 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:21.029298 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:21.921090 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:21.921051 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:22.622258 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:22.622220 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:23.511937 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:23.511905 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:24.326358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:24.326324 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:25.150227 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:25.150193 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:26.115795 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:26.115757 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:26.853868 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:13:26.853836 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:13:27.117365 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:27.117279 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:27.726152 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:27.726119 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:28.603706 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:28.603669 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:29.429036 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:29.428995 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:30.203911 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:30.203864 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:31.117425 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:31.117385 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:32.111350 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:32.111316 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:32.930184 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:32.930155 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:33.915062 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:33.915025 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:34.790639 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:34.790602 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:35.605023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:35.604990 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:36.457155 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:36.457101 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:37.305775 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:37.305742 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:38.013081 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:38.013049 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:38.912851 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:38.912821 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:39.785148 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:39.785113 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:40.516052 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:40.516013 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:41.410816 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:41.410776 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:42.230117 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:42.230075 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:42.552596 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:42.552511 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:13:42.975261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:42.975228 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:43.814929 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:43.814873 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:44.728492 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:44.728458 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:45.518387 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:45.518355 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:46.521055 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:46.521024 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:47.406166 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:47.406130 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:48.195960 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:48.195922 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:48.765765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:48.765739 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-sbd6p_fbe0b6eb-e661-4ac4-ac42-ac13e163f5de/payload-processing/0.log" Nov 02 10:13:48.944441 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:48.944412 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-sbd6p_fbe0b6eb-e661-4ac4-ac42-ac13e163f5de/payload-processing/0.log" Nov 02 10:13:49.104188 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:49.104152 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:50.096245 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:50.096218 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:50.271312 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:50.271280 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:50.817016 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:50.816982 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:51.808914 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:51.808853 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:52.722370 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:52.722319 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:53.433014 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:53.432979 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:54.310512 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:54.310481 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:55.201459 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:55.201430 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:55.378295 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:55.378257 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:55.918994 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:55.918954 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:56.818796 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:56.818731 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:56.853610 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:13:56.853575 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:13:57.721444 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:57.721413 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:58.424573 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:58.424538 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:59.138745 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:59.138709 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:13:59.424476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:13:59.424390 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:00.315333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:00.315303 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:01.022125 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:01.022087 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:01.911111 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:01.911077 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:02.800261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:02.800229 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:03.519543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:03.519513 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:04.426703 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:04.426667 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:05.309683 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:05.309641 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:06.128777 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:06.128735 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:07.000858 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:07.000824 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:07.883801 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:07.883765 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:08.610679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:08.610644 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:09.516115 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:09.516076 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:10.322471 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:10.322435 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:11.118445 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:11.118408 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:12.023112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:12.023077 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:12.519778 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:12.519743 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:12.836199 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:12.836170 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:13.613250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:13.613213 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:14.611254 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:14.611224 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:15.667487 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:15.667453 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:16.417613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:16.417584 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:17.324023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:17.323985 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:18.310734 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:18.310696 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:19.013433 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:19.013398 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:19.912546 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:19.912510 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:20.795502 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:20.795472 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:21.624024 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:21.623986 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:22.505277 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:22.505241 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:23.421570 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:23.421541 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:24.122823 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:24.122790 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:25.029700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:25.029663 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:25.919544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:25.919505 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:26.627362 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:26.627329 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:26.854918 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:14:26.854866 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:14:27.530454 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:27.530424 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:28.425167 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:28.425130 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:29.180517 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:29.180487 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:30.006876 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:30.006844 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:30.810505 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:30.810459 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:31.515051 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:31.515003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:31.894006 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:31.893976 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:32.890813 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:32.890770 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:33.352164 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:33.352125 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:33.846122 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:33.846089 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:34.402216 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:34.402177 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:14:34.683886 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:34.683804 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:35.717765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:35.717736 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:36.637744 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:36.637711 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:37.614515 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:37.614474 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:38.316869 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:38.316832 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:39.208721 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:39.208679 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:39.948433 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:39.948399 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:40.136124 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:40.136090 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:40.830607 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:40.830561 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:41.623330 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:41.623274 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:42.527038 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:42.526991 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:43.239677 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:43.239646 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:44.137107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:44.137043 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:44.928982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:44.928948 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:45.813204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:45.813166 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:46.308783 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:46.308750 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:46.545708 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:46.545678 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:47.515625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:47.515583 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:47.701382 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:47.701353 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:47.999712 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:47.999678 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6"] Nov 02 10:14:48.000004 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:47.999969 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" podUID="41a920f9-c8b1-4129-84f7-d206d320d833" containerName="maas-api" containerID="cri-o://b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd" gracePeriod=30 Nov 02 10:14:48.090937 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.090901 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk"] Nov 02 10:14:48.091253 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.091191 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" podUID="0ca46343-303c-4d67-a855-667123794849" containerName="payload-pre-processing" containerID="cri-o://90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8" gracePeriod=30 Nov 02 10:14:48.141701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.141653 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:48.312123 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.310809 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:14:48.329403 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.329353 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:48.365135 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.365103 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-metrics-certs\") pod \"41a920f9-c8b1-4129-84f7-d206d320d833\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " Nov 02 10:14:48.365352 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.365207 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a920f9-c8b1-4129-84f7-d206d320d833-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a920f9-c8b1-4129-84f7-d206d320d833-openshift-service-ca\") pod \"41a920f9-c8b1-4129-84f7-d206d320d833\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " Nov 02 10:14:48.365352 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.365267 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls\") pod \"41a920f9-c8b1-4129-84f7-d206d320d833\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " Nov 02 10:14:48.365352 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.365310 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a920f9-c8b1-4129-84f7-d206d320d833-kube-api-access-dmcwf\" (UniqueName: \"kubernetes.io/projected/41a920f9-c8b1-4129-84f7-d206d320d833-kube-api-access-dmcwf\") pod \"41a920f9-c8b1-4129-84f7-d206d320d833\" (UID: \"41a920f9-c8b1-4129-84f7-d206d320d833\") " Nov 02 10:14:48.365691 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.365665 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a920f9-c8b1-4129-84f7-d206d320d833-openshift-service-ca" pod "41a920f9-c8b1-4129-84f7-d206d320d833" (UID: "41a920f9-c8b1-4129-84f7-d206d320d833"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:14:48.367833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.367806 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-metrics-certs" pod "41a920f9-c8b1-4129-84f7-d206d320d833" (UID: "41a920f9-c8b1-4129-84f7-d206d320d833"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:14:48.368112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.368091 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls" pod "41a920f9-c8b1-4129-84f7-d206d320d833" (UID: "41a920f9-c8b1-4129-84f7-d206d320d833"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:14:48.369906 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.369853 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a920f9-c8b1-4129-84f7-d206d320d833-kube-api-access-dmcwf" pod "41a920f9-c8b1-4129-84f7-d206d320d833" (UID: "41a920f9-c8b1-4129-84f7-d206d320d833"). InnerVolumeSpecName "kube-api-access-dmcwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:14:48.387148 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.387119 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:14:48.426448 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.426395 4467 generic.go:356] "Generic (PLEG): container finished" podID="0ca46343-303c-4d67-a855-667123794849" containerID="90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8" exitCode=0 Nov 02 10:14:48.426448 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.426430 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" event={"ID":"0ca46343-303c-4d67-a855-667123794849","Type":"ContainerDied","Data":"90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8"} Nov 02 10:14:48.426709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.426506 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" Nov 02 10:14:48.426709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.426524 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk" event={"ID":"0ca46343-303c-4d67-a855-667123794849","Type":"ContainerDied","Data":"e44e3439311bae4ecf1578b2c693e09824f733f2dd988341613869461a0c4092"} Nov 02 10:14:48.426709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.426555 4467 scope.go:122] "RemoveContainer" containerID="90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8" Nov 02 10:14:48.429197 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.429169 4467 generic.go:356] "Generic (PLEG): container finished" podID="41a920f9-c8b1-4129-84f7-d206d320d833" containerID="b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd" exitCode=0 Nov 02 10:14:48.429328 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.429229 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" Nov 02 10:14:48.429328 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.429275 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" event={"ID":"41a920f9-c8b1-4129-84f7-d206d320d833","Type":"ContainerDied","Data":"b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd"} Nov 02 10:14:48.429328 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.429315 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6" event={"ID":"41a920f9-c8b1-4129-84f7-d206d320d833","Type":"ContainerDied","Data":"19b783ffbce0576a4a41d7cf0879d4276e98f420dcfa729b40cb4a8a0290c2bf"} Nov 02 10:14:48.436953 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.436933 4467 scope.go:122] "RemoveContainer" containerID="90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8" Nov 02 10:14:48.437270 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:14:48.437241 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8\": container with ID starting with 90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8 not found: ID does not exist" containerID="90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8" Nov 02 10:14:48.437384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.437279 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8"} err="failed to get container status \"90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8\": rpc error: code = NotFound desc = could not find container \"90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8\": container with ID starting with 90b58adb2bf48ee459894065cfdc0531967af056ea051f49c68ec9be2d3e5ca8 not found: ID does not exist" Nov 02 10:14:48.437384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.437305 4467 scope.go:122] "RemoveContainer" containerID="b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd" Nov 02 10:14:48.449109 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.449089 4467 scope.go:122] "RemoveContainer" containerID="b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd" Nov 02 10:14:48.452416 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:14:48.452386 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd\": container with ID starting with b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd not found: ID does not exist" containerID="b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd" Nov 02 10:14:48.452524 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.452422 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd"} err="failed to get container status \"b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd\": rpc error: code = NotFound desc = could not find container \"b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd\": container with ID starting with b1c26c8320aad4cc14b51293e176237512ceab915d4286e0a3285d855e41b9cd not found: ID does not exist" Nov 02 10:14:48.454129 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.454108 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6"] Nov 02 10:14:48.457883 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.457857 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2b5a18-68577d6cb5-zq9t6"] Nov 02 10:14:48.466963 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.466935 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca46343-303c-4d67-a855-667123794849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca46343-303c-4d67-a855-667123794849-plugins-config-volume\") pod \"0ca46343-303c-4d67-a855-667123794849\" (UID: \"0ca46343-303c-4d67-a855-667123794849\") " Nov 02 10:14:48.467069 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.467003 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca46343-303c-4d67-a855-667123794849-kube-api-access-26lzc\" (UniqueName: \"kubernetes.io/projected/0ca46343-303c-4d67-a855-667123794849-kube-api-access-26lzc\") pod \"0ca46343-303c-4d67-a855-667123794849\" (UID: \"0ca46343-303c-4d67-a855-667123794849\") " Nov 02 10:14:48.467359 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.467336 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca46343-303c-4d67-a855-667123794849-plugins-config-volume" pod "0ca46343-303c-4d67-a855-667123794849" (UID: "0ca46343-303c-4d67-a855-667123794849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:14:48.467397 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.467360 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:14:48.467397 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.467384 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmcwf\" (UniqueName: \"kubernetes.io/projected/41a920f9-c8b1-4129-84f7-d206d320d833-kube-api-access-dmcwf\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:14:48.467466 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.467399 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a920f9-c8b1-4129-84f7-d206d320d833-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:14:48.467466 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.467409 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a920f9-c8b1-4129-84f7-d206d320d833-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:14:48.469304 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.469283 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca46343-303c-4d67-a855-667123794849-kube-api-access-26lzc" pod "0ca46343-303c-4d67-a855-667123794849" (UID: "0ca46343-303c-4d67-a855-667123794849"). InnerVolumeSpecName "kube-api-access-26lzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:14:48.567943 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.567910 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca46343-303c-4d67-a855-667123794849-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:14:48.567943 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.567940 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26lzc\" (UniqueName: \"kubernetes.io/projected/0ca46343-303c-4d67-a855-667123794849-kube-api-access-26lzc\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:14:48.599635 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.599603 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:48.751048 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.751003 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk"] Nov 02 10:14:48.756808 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:48.756777 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2b5a18-85d6575455-fbmjk"] Nov 02 10:14:49.331727 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:49.331689 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca46343-303c-4d67-a855-667123794849" path="/var/lib/kubelet/pods/0ca46343-303c-4d67-a855-667123794849/volumes" Nov 02 10:14:49.332122 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:49.332106 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a920f9-c8b1-4129-84f7-d206d320d833" path="/var/lib/kubelet/pods/41a920f9-c8b1-4129-84f7-d206d320d833/volumes" Nov 02 10:14:49.509358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:49.509321 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:49.691789 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:49.691707 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:14:50.311146 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:50.311110 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:51.133086 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:51.133052 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:51.810844 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:51.810806 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:52.730414 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:52.730378 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:53.159783 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:53.159749 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:54.461131 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:54.461092 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:55.324078 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:55.324039 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:56.330035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:56.330004 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:56.854546 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:14:56.854510 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:14:57.052768 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:57.052718 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:58.112450 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:58.112415 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:58.827183 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:58.827150 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:14:59.836909 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:14:59.836864 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:00.378738 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:00.378688 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:00.600628 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:00.600597 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:01.507864 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:01.507832 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:02.338712 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:02.338667 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:02.611511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:02.611434 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:02.787951 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:02.787912 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:03.219704 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:03.219658 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:04.016710 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:04.016675 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:04.904116 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:04.904079 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:05.616758 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:05.616722 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:06.738515 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:06.738476 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:06.949544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:06.949505 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:07.284671 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.284636 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8"] Nov 02 10:15:07.284953 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.284929 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" podUID="5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" containerName="maas-api" containerID="cri-o://936a202176da997f4490765f491d0210d1907d7823937b4b2808de4c84bf9034" gracePeriod=30 Nov 02 10:15:07.373921 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.373857 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4"] Nov 02 10:15:07.374167 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.374137 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" podUID="879f2fd5-92ac-473f-8b10-f66c49ec445a" containerName="payload-pre-processing" containerID="cri-o://7e69ace79c599e746f95f118b3ed20217bcefdd6fcc2f467a7e5c8764f568ec9" gracePeriod=30 Nov 02 10:15:07.424494 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.424371 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:07.535670 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.535612 4467 generic.go:356] "Generic (PLEG): container finished" podID="5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" containerID="936a202176da997f4490765f491d0210d1907d7823937b4b2808de4c84bf9034" exitCode=0 Nov 02 10:15:07.535785 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.535683 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" event={"ID":"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70","Type":"ContainerDied","Data":"936a202176da997f4490765f491d0210d1907d7823937b4b2808de4c84bf9034"} Nov 02 10:15:07.538602 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.538579 4467 generic.go:356] "Generic (PLEG): container finished" podID="879f2fd5-92ac-473f-8b10-f66c49ec445a" containerID="7e69ace79c599e746f95f118b3ed20217bcefdd6fcc2f467a7e5c8764f568ec9" exitCode=0 Nov 02 10:15:07.538728 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.538629 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" event={"ID":"879f2fd5-92ac-473f-8b10-f66c49ec445a","Type":"ContainerDied","Data":"7e69ace79c599e746f95f118b3ed20217bcefdd6fcc2f467a7e5c8764f568ec9"} Nov 02 10:15:07.584378 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.584341 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:15:07.603863 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.603839 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:07.659451 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.659416 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-openshift-service-ca\") pod \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " Nov 02 10:15:07.659648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.659511 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-maas-api-tls\") pod \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " Nov 02 10:15:07.659648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.659595 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-kube-api-access-5tdq5\" (UniqueName: \"kubernetes.io/projected/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-kube-api-access-5tdq5\") pod \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " Nov 02 10:15:07.659648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.659624 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-metrics-certs\") pod \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\" (UID: \"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70\") " Nov 02 10:15:07.659862 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.659838 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-openshift-service-ca" pod "5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" (UID: "5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:15:07.661686 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.661657 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-maas-api-tls" pod "5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" (UID: "5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:15:07.661978 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.661954 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-kube-api-access-5tdq5" pod "5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" (UID: "5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70"). InnerVolumeSpecName "kube-api-access-5tdq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:15:07.662196 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.662172 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-metrics-certs" pod "5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" (UID: "5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:15:07.678588 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.678567 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:15:07.760110 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.760081 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879f2fd5-92ac-473f-8b10-f66c49ec445a-kube-api-access-c4mrh\" (UniqueName: \"kubernetes.io/projected/879f2fd5-92ac-473f-8b10-f66c49ec445a-kube-api-access-c4mrh\") pod \"879f2fd5-92ac-473f-8b10-f66c49ec445a\" (UID: \"879f2fd5-92ac-473f-8b10-f66c49ec445a\") " Nov 02 10:15:07.760511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.760193 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879f2fd5-92ac-473f-8b10-f66c49ec445a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f2fd5-92ac-473f-8b10-f66c49ec445a-plugins-config-volume\") pod \"879f2fd5-92ac-473f-8b10-f66c49ec445a\" (UID: \"879f2fd5-92ac-473f-8b10-f66c49ec445a\") " Nov 02 10:15:07.760511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.760421 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:15:07.760511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.760436 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tdq5\" (UniqueName: \"kubernetes.io/projected/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-kube-api-access-5tdq5\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:15:07.760511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.760446 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:15:07.760511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.760454 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:15:07.760693 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.760610 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879f2fd5-92ac-473f-8b10-f66c49ec445a-plugins-config-volume" pod "879f2fd5-92ac-473f-8b10-f66c49ec445a" (UID: "879f2fd5-92ac-473f-8b10-f66c49ec445a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:15:07.762156 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.762135 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879f2fd5-92ac-473f-8b10-f66c49ec445a-kube-api-access-c4mrh" pod "879f2fd5-92ac-473f-8b10-f66c49ec445a" (UID: "879f2fd5-92ac-473f-8b10-f66c49ec445a"). InnerVolumeSpecName "kube-api-access-c4mrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:15:07.861850 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.861817 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4mrh\" (UniqueName: \"kubernetes.io/projected/879f2fd5-92ac-473f-8b10-f66c49ec445a-kube-api-access-c4mrh\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:15:07.861850 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:07.861853 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f2fd5-92ac-473f-8b10-f66c49ec445a-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:15:08.545532 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.545502 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" Nov 02 10:15:08.545735 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.545535 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8" event={"ID":"5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70","Type":"ContainerDied","Data":"f7aedddf18a37682cbe3074ba992a765923c1b225d2947e3375e7b9cda7da84a"} Nov 02 10:15:08.545735 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.545595 4467 scope.go:122] "RemoveContainer" containerID="936a202176da997f4490765f491d0210d1907d7823937b4b2808de4c84bf9034" Nov 02 10:15:08.546936 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.546915 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" event={"ID":"879f2fd5-92ac-473f-8b10-f66c49ec445a","Type":"ContainerDied","Data":"327a55eba0a505a4038c468a8476c84f8862e5eb9ca746fbd9ee2e4dc1723cfa"} Nov 02 10:15:08.546994 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.546961 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4" Nov 02 10:15:08.554768 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.554751 4467 scope.go:122] "RemoveContainer" containerID="7e69ace79c599e746f95f118b3ed20217bcefdd6fcc2f467a7e5c8764f568ec9" Nov 02 10:15:08.573084 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.573060 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8"] Nov 02 10:15:08.577209 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.577189 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-836bba-6b65597869-fbkv8"] Nov 02 10:15:08.589578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.589546 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4"] Nov 02 10:15:08.594840 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.594819 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-836bba-6bb8db78bc-l4vc4"] Nov 02 10:15:08.873706 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:08.873666 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:15:09.334047 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:09.334016 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" path="/var/lib/kubelet/pods/5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70/volumes" Nov 02 10:15:09.334402 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:09.334389 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879f2fd5-92ac-473f-8b10-f66c49ec445a" path="/var/lib/kubelet/pods/879f2fd5-92ac-473f-8b10-f66c49ec445a/volumes" Nov 02 10:15:12.591061 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:12.591011 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:12.781133 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:12.781098 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:13.096207 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:13.096169 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:13.469150 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:13.469059 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:14.921101 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921064 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l"] Nov 02 10:15:14.921485 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921476 4467 cpu_manager.go:414] "containerMap: removing container" podUID="0ca46343-303c-4d67-a855-667123794849" containerName="payload-pre-processing" Nov 02 10:15:14.921525 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921488 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca46343-303c-4d67-a855-667123794849" containerName="payload-pre-processing" Nov 02 10:15:14.921525 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921498 4467 cpu_manager.go:414] "containerMap: removing container" podUID="05817456-2b7f-4e38-8e2d-2ef0aa88f1db" containerName="payload-processing" Nov 02 10:15:14.921525 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921504 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05817456-2b7f-4e38-8e2d-2ef0aa88f1db" containerName="payload-processing" Nov 02 10:15:14.921525 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921518 4467 cpu_manager.go:414] "containerMap: removing container" podUID="41a920f9-c8b1-4129-84f7-d206d320d833" containerName="maas-api" Nov 02 10:15:14.921525 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921523 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a920f9-c8b1-4129-84f7-d206d320d833" containerName="maas-api" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921542 4467 cpu_manager.go:414] "containerMap: removing container" podUID="998685e9-4460-4bc6-8bd0-a198d7229d21" containerName="payload-pre-processing" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921547 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998685e9-4460-4bc6-8bd0-a198d7229d21" containerName="payload-pre-processing" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921555 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" containerName="maas-api" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921560 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" containerName="maas-api" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921566 4467 cpu_manager.go:414] "containerMap: removing container" podUID="879f2fd5-92ac-473f-8b10-f66c49ec445a" containerName="payload-pre-processing" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921570 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879f2fd5-92ac-473f-8b10-f66c49ec445a" containerName="payload-pre-processing" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921630 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca46343-303c-4d67-a855-667123794849" containerName="payload-pre-processing" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921638 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="05817456-2b7f-4e38-8e2d-2ef0aa88f1db" containerName="payload-processing" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921645 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a920f9-c8b1-4129-84f7-d206d320d833" containerName="maas-api" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921654 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="998685e9-4460-4bc6-8bd0-a198d7229d21" containerName="payload-pre-processing" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921661 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5060c7c4-bdf2-46d1-92e8-a1b3ecd96d70" containerName="maas-api" Nov 02 10:15:14.921701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.921666 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="879f2fd5-92ac-473f-8b10-f66c49ec445a" containerName="payload-pre-processing" Nov 02 10:15:14.926333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.926316 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:14.933809 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:14.933782 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l"] Nov 02 10:15:15.031967 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.031924 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgsv\" (UniqueName: \"kubernetes.io/projected/0f571c9e-8773-4619-a170-a3415e2d3095-kube-api-access-tjgsv\") pod \"payload-pre-processing-6c45b49d64-2dd4l\" (UID: \"0f571c9e-8773-4619-a170-a3415e2d3095\") " pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:15.032146 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.031979 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f571c9e-8773-4619-a170-a3415e2d3095-plugins-config-volume\") pod \"payload-pre-processing-6c45b49d64-2dd4l\" (UID: \"0f571c9e-8773-4619-a170-a3415e2d3095\") " pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:15.132597 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.132561 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgsv\" (UniqueName: \"kubernetes.io/projected/0f571c9e-8773-4619-a170-a3415e2d3095-kube-api-access-tjgsv\") pod \"payload-pre-processing-6c45b49d64-2dd4l\" (UID: \"0f571c9e-8773-4619-a170-a3415e2d3095\") " pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:15.132763 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.132604 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f571c9e-8773-4619-a170-a3415e2d3095-plugins-config-volume\") pod \"payload-pre-processing-6c45b49d64-2dd4l\" (UID: \"0f571c9e-8773-4619-a170-a3415e2d3095\") " pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:15.133315 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.133296 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f571c9e-8773-4619-a170-a3415e2d3095-plugins-config-volume\") pod \"payload-pre-processing-6c45b49d64-2dd4l\" (UID: \"0f571c9e-8773-4619-a170-a3415e2d3095\") " pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:15.145961 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.145930 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgsv\" (UniqueName: \"kubernetes.io/projected/0f571c9e-8773-4619-a170-a3415e2d3095-kube-api-access-tjgsv\") pod \"payload-pre-processing-6c45b49d64-2dd4l\" (UID: \"0f571c9e-8773-4619-a170-a3415e2d3095\") " pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:15.240262 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.240192 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:15.513982 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.513956 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l"] Nov 02 10:15:15.516679 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:15:15.516645 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f571c9e_8773_4619_a170_a3415e2d3095.slice/crio-50ac701a7154b6d5b21c442b3c824efc7e210c13c0130dcb49b09570b95901bd WatchSource:0}: Error finding container 50ac701a7154b6d5b21c442b3c824efc7e210c13c0130dcb49b09570b95901bd: Status 404 returned error can't find the container with id 50ac701a7154b6d5b21c442b3c824efc7e210c13c0130dcb49b09570b95901bd Nov 02 10:15:15.590705 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:15.590677 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" event={"ID":"0f571c9e-8773-4619-a170-a3415e2d3095","Type":"ContainerStarted","Data":"50ac701a7154b6d5b21c442b3c824efc7e210c13c0130dcb49b09570b95901bd"} Nov 02 10:15:16.597336 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:16.597297 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" event={"ID":"0f571c9e-8773-4619-a170-a3415e2d3095","Type":"ContainerStarted","Data":"2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c"} Nov 02 10:15:16.597861 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:16.597413 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:16.618154 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:16.618102 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" podStartSLOduration=2.618090468 podStartE2EDuration="2.618090468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:15:16.616031528 +0000 UTC m=+2395.455973746" watchObservedRunningTime="2026-11-02 10:15:16.618090468 +0000 UTC m=+2395.458032462" Nov 02 10:15:26.292219 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:26.292180 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:26.855397 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:15:26.855363 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:15:27.605651 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:27.605618 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:15:29.427332 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.427285 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-76459f68c5-48rr8"] Nov 02 10:15:29.432554 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.432530 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:29.439169 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.439135 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-76459f68c5-48rr8"] Nov 02 10:15:29.453440 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.453405 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4028289-efd5-4100-8594-85c937a4bdc1-plugins-config-volume\") pod \"payload-processing-76459f68c5-48rr8\" (UID: \"d4028289-efd5-4100-8594-85c937a4bdc1\") " pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:29.453440 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.453444 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kx76\" (UniqueName: \"kubernetes.io/projected/d4028289-efd5-4100-8594-85c937a4bdc1-kube-api-access-5kx76\") pod \"payload-processing-76459f68c5-48rr8\" (UID: \"d4028289-efd5-4100-8594-85c937a4bdc1\") " pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:29.554651 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.554615 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4028289-efd5-4100-8594-85c937a4bdc1-plugins-config-volume\") pod \"payload-processing-76459f68c5-48rr8\" (UID: \"d4028289-efd5-4100-8594-85c937a4bdc1\") " pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:29.554651 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.554652 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kx76\" (UniqueName: \"kubernetes.io/projected/d4028289-efd5-4100-8594-85c937a4bdc1-kube-api-access-5kx76\") pod \"payload-processing-76459f68c5-48rr8\" (UID: \"d4028289-efd5-4100-8594-85c937a4bdc1\") " pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:29.555373 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.555350 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4028289-efd5-4100-8594-85c937a4bdc1-plugins-config-volume\") pod \"payload-processing-76459f68c5-48rr8\" (UID: \"d4028289-efd5-4100-8594-85c937a4bdc1\") " pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:29.569192 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.569167 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kx76\" (UniqueName: \"kubernetes.io/projected/d4028289-efd5-4100-8594-85c937a4bdc1-kube-api-access-5kx76\") pod \"payload-processing-76459f68c5-48rr8\" (UID: \"d4028289-efd5-4100-8594-85c937a4bdc1\") " pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:29.751134 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:29.751053 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:30.027424 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:30.027399 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-76459f68c5-48rr8"] Nov 02 10:15:30.030012 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:15:30.029985 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4028289_efd5_4100_8594_85c937a4bdc1.slice/crio-4b99226ae2e2be4b016b18cf205404a494057d2a66ba5912916c79a1cd7041c5 WatchSource:0}: Error finding container 4b99226ae2e2be4b016b18cf205404a494057d2a66ba5912916c79a1cd7041c5: Status 404 returned error can't find the container with id 4b99226ae2e2be4b016b18cf205404a494057d2a66ba5912916c79a1cd7041c5 Nov 02 10:15:30.670407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:30.670375 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" event={"ID":"d4028289-efd5-4100-8594-85c937a4bdc1","Type":"ContainerStarted","Data":"19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d"} Nov 02 10:15:30.670837 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:30.670426 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" event={"ID":"d4028289-efd5-4100-8594-85c937a4bdc1","Type":"ContainerStarted","Data":"4b99226ae2e2be4b016b18cf205404a494057d2a66ba5912916c79a1cd7041c5"} Nov 02 10:15:30.670837 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:30.670471 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:30.691495 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:30.691441 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" podStartSLOduration=1.6914273199999998 podStartE2EDuration="1.69142732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:15:30.689263608 +0000 UTC m=+2409.529205604" watchObservedRunningTime="2026-11-02 10:15:30.69142732 +0000 UTC m=+2409.531369314" Nov 02 10:15:41.678367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:41.678332 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:15:41.727189 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:41.727144 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sbd6p"] Nov 02 10:15:43.857233 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:43.857197 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68"] Nov 02 10:15:43.861140 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:43.861118 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:43.870400 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:43.870371 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68"] Nov 02 10:15:43.987571 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:43.987532 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcz4w\" (UniqueName: \"kubernetes.io/projected/6fc09a83-a593-4e4e-b4c0-4921265e7ac1-kube-api-access-wcz4w\") pod \"payload-pre-processing-6dbbcf55d8-n4d68\" (UID: \"6fc09a83-a593-4e4e-b4c0-4921265e7ac1\") " pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:43.987763 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:43.987727 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc09a83-a593-4e4e-b4c0-4921265e7ac1-plugins-config-volume\") pod \"payload-pre-processing-6dbbcf55d8-n4d68\" (UID: \"6fc09a83-a593-4e4e-b4c0-4921265e7ac1\") " pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:44.089068 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.089022 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc09a83-a593-4e4e-b4c0-4921265e7ac1-plugins-config-volume\") pod \"payload-pre-processing-6dbbcf55d8-n4d68\" (UID: \"6fc09a83-a593-4e4e-b4c0-4921265e7ac1\") " pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:44.089256 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.089085 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcz4w\" (UniqueName: \"kubernetes.io/projected/6fc09a83-a593-4e4e-b4c0-4921265e7ac1-kube-api-access-wcz4w\") pod \"payload-pre-processing-6dbbcf55d8-n4d68\" (UID: \"6fc09a83-a593-4e4e-b4c0-4921265e7ac1\") " pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:44.089757 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.089736 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc09a83-a593-4e4e-b4c0-4921265e7ac1-plugins-config-volume\") pod \"payload-pre-processing-6dbbcf55d8-n4d68\" (UID: \"6fc09a83-a593-4e4e-b4c0-4921265e7ac1\") " pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:44.102810 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.102779 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcz4w\" (UniqueName: \"kubernetes.io/projected/6fc09a83-a593-4e4e-b4c0-4921265e7ac1-kube-api-access-wcz4w\") pod \"payload-pre-processing-6dbbcf55d8-n4d68\" (UID: \"6fc09a83-a593-4e4e-b4c0-4921265e7ac1\") " pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:44.176618 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.176524 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:44.452679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.452617 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68"] Nov 02 10:15:44.454985 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:15:44.454954 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc09a83_a593_4e4e_b4c0_4921265e7ac1.slice/crio-05074e25cff9b4c4451b8936a69fc0e5b78e941eaabd3c4f39b12d198e3039ca WatchSource:0}: Error finding container 05074e25cff9b4c4451b8936a69fc0e5b78e941eaabd3c4f39b12d198e3039ca: Status 404 returned error can't find the container with id 05074e25cff9b4c4451b8936a69fc0e5b78e941eaabd3c4f39b12d198e3039ca Nov 02 10:15:44.742643 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.742548 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" event={"ID":"6fc09a83-a593-4e4e-b4c0-4921265e7ac1","Type":"ContainerStarted","Data":"79982f5b608cb8daaa99d0801e170c33e6a0ee11d5bc98338351dce6c9dabbee"} Nov 02 10:15:44.742643 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.742615 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" event={"ID":"6fc09a83-a593-4e4e-b4c0-4921265e7ac1","Type":"ContainerStarted","Data":"05074e25cff9b4c4451b8936a69fc0e5b78e941eaabd3c4f39b12d198e3039ca"} Nov 02 10:15:44.742880 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.742761 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:44.764035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:44.763974 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" podStartSLOduration=1.763958508 podStartE2EDuration="1.763958508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:15:44.761578086 +0000 UTC m=+2423.601520081" watchObservedRunningTime="2026-11-02 10:15:44.763958508 +0000 UTC m=+2423.603900502" Nov 02 10:15:46.727790 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:46.727733 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" podUID="fbe0b6eb-e661-4ac4-ac42-ac13e163f5de" containerName="payload-processing" containerID="cri-o://db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8" gracePeriod=25 Nov 02 10:15:46.983400 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:46.983331 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 10:15:47.120862 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.120817 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-plugins-config-volume\") pod \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\" (UID: \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\") " Nov 02 10:15:47.121066 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.120962 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-kube-api-access-lnlt5\" (UniqueName: \"kubernetes.io/projected/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-kube-api-access-lnlt5\") pod \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\" (UID: \"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de\") " Nov 02 10:15:47.121222 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.121198 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-plugins-config-volume" pod "fbe0b6eb-e661-4ac4-ac42-ac13e163f5de" (UID: "fbe0b6eb-e661-4ac4-ac42-ac13e163f5de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:15:47.123002 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.122972 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-kube-api-access-lnlt5" pod "fbe0b6eb-e661-4ac4-ac42-ac13e163f5de" (UID: "fbe0b6eb-e661-4ac4-ac42-ac13e163f5de"). InnerVolumeSpecName "kube-api-access-lnlt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:15:47.222314 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.222274 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:15:47.222314 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.222307 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnlt5\" (UniqueName: \"kubernetes.io/projected/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de-kube-api-access-lnlt5\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:15:47.761915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.761811 4467 generic.go:356] "Generic (PLEG): container finished" podID="fbe0b6eb-e661-4ac4-ac42-ac13e163f5de" containerID="db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8" exitCode=0 Nov 02 10:15:47.761915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.761851 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" Nov 02 10:15:47.761915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.761901 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" event={"ID":"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de","Type":"ContainerDied","Data":"db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8"} Nov 02 10:15:47.762483 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.761979 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-sbd6p" event={"ID":"fbe0b6eb-e661-4ac4-ac42-ac13e163f5de","Type":"ContainerDied","Data":"81125318615b20dfb6c1e69718bb0ee287299503bbaa012268a5d68135c25ded"} Nov 02 10:15:47.762483 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.762015 4467 scope.go:122] "RemoveContainer" containerID="db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8" Nov 02 10:15:47.771779 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.771760 4467 scope.go:122] "RemoveContainer" containerID="db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8" Nov 02 10:15:47.772072 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:15:47.772053 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8\": container with ID starting with db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8 not found: ID does not exist" containerID="db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8" Nov 02 10:15:47.772132 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.772081 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8"} err="failed to get container status \"db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8\": rpc error: code = NotFound desc = could not find container \"db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8\": container with ID starting with db96068bd60f9ebef107865ddf13a0eacda94f580ca3a57485da62e6dd5d89c8 not found: ID does not exist" Nov 02 10:15:47.781052 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.781026 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sbd6p"] Nov 02 10:15:47.785389 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:47.785361 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sbd6p"] Nov 02 10:15:49.331297 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:49.331261 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe0b6eb-e661-4ac4-ac42-ac13e163f5de" path="/var/lib/kubelet/pods/fbe0b6eb-e661-4ac4-ac42-ac13e163f5de/volumes" Nov 02 10:15:55.752129 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:55.752096 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6dbbcf55d8-n4d68" Nov 02 10:15:55.778711 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:55.778672 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-76459f68c5-48rr8"] Nov 02 10:15:55.800659 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:55.800625 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l"] Nov 02 10:15:56.854521 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:15:56.854487 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:15:58.128707 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.128592 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:58.395265 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.395158 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:58.642499 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.642441 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:15:58.673490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.673409 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9"] Nov 02 10:15:58.673900 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.673877 4467 cpu_manager.go:414] "containerMap: removing container" podUID="fbe0b6eb-e661-4ac4-ac42-ac13e163f5de" containerName="payload-processing" Nov 02 10:15:58.673955 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.673908 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe0b6eb-e661-4ac4-ac42-ac13e163f5de" containerName="payload-processing" Nov 02 10:15:58.673992 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.673982 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe0b6eb-e661-4ac4-ac42-ac13e163f5de" containerName="payload-processing" Nov 02 10:15:58.677300 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.677284 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.690299 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.690272 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 10:15:58.691116 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.691089 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 10:15:58.691252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.691120 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 02 10:15:58.691252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.691122 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 10:15:58.691252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.691148 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5r69g\"" Nov 02 10:15:58.691252 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.691120 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 10:15:58.698520 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.698492 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9"] Nov 02 10:15:58.843073 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.843037 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.843277 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.843100 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tx5l\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-kube-api-access-4tx5l\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.843277 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.843173 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.843277 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.843266 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.944777 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.944689 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.944777 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.944741 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.945007 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.944801 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tx5l\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-kube-api-access-4tx5l\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.945007 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.944870 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.945552 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.945520 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.945693 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.945673 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.947233 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.947212 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.957776 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.957744 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tx5l\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-kube-api-access-4tx5l\") pod \"payload-processing-external-model-5bcd7997-kt9n9\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:58.990823 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:58.990790 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:59.275273 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:59.275245 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9"] Nov 02 10:15:59.277145 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:15:59.277114 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c26b21_7d51_4d0e_859a_bb4a7844881e.slice/crio-c6c8a7a4cdede9cab2882c476bd803d57d943df0812405c0f1b7f5644793b67f WatchSource:0}: Error finding container c6c8a7a4cdede9cab2882c476bd803d57d943df0812405c0f1b7f5644793b67f: Status 404 returned error can't find the container with id c6c8a7a4cdede9cab2882c476bd803d57d943df0812405c0f1b7f5644793b67f Nov 02 10:15:59.827776 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:59.827734 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" event={"ID":"d3c26b21-7d51-4d0e-859a-bb4a7844881e","Type":"ContainerStarted","Data":"07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab"} Nov 02 10:15:59.828013 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:59.827792 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" event={"ID":"d3c26b21-7d51-4d0e-859a-bb4a7844881e","Type":"ContainerStarted","Data":"c6c8a7a4cdede9cab2882c476bd803d57d943df0812405c0f1b7f5644793b67f"} Nov 02 10:15:59.828013 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:59.827819 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:15:59.848227 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:59.848171 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" podStartSLOduration=1.8481563699999999 podStartE2EDuration="1.84815637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:15:59.846466057 +0000 UTC m=+2438.686408055" watchObservedRunningTime="2026-11-02 10:15:59.84815637 +0000 UTC m=+2438.688098364" Nov 02 10:15:59.883666 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:15:59.883618 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:16:00.779137 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:00.779071 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" podUID="d4028289-efd5-4100-8594-85c937a4bdc1" containerName="payload-processing" containerID="cri-o://19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d" gracePeriod=25 Nov 02 10:16:00.801461 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:00.801416 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" podUID="0f571c9e-8773-4619-a170-a3415e2d3095" containerName="payload-pre-processing" containerID="cri-o://2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c" gracePeriod=25 Nov 02 10:16:01.114550 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.114516 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:16:01.159839 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.159806 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:16:01.271570 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.271517 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f571c9e-8773-4619-a170-a3415e2d3095-kube-api-access-tjgsv\" (UniqueName: \"kubernetes.io/projected/0f571c9e-8773-4619-a170-a3415e2d3095-kube-api-access-tjgsv\") pod \"0f571c9e-8773-4619-a170-a3415e2d3095\" (UID: \"0f571c9e-8773-4619-a170-a3415e2d3095\") " Nov 02 10:16:01.271757 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.271599 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4028289-efd5-4100-8594-85c937a4bdc1-kube-api-access-5kx76\" (UniqueName: \"kubernetes.io/projected/d4028289-efd5-4100-8594-85c937a4bdc1-kube-api-access-5kx76\") pod \"d4028289-efd5-4100-8594-85c937a4bdc1\" (UID: \"d4028289-efd5-4100-8594-85c937a4bdc1\") " Nov 02 10:16:01.271757 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.271658 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4028289-efd5-4100-8594-85c937a4bdc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4028289-efd5-4100-8594-85c937a4bdc1-plugins-config-volume\") pod \"d4028289-efd5-4100-8594-85c937a4bdc1\" (UID: \"d4028289-efd5-4100-8594-85c937a4bdc1\") " Nov 02 10:16:01.271757 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.271703 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f571c9e-8773-4619-a170-a3415e2d3095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f571c9e-8773-4619-a170-a3415e2d3095-plugins-config-volume\") pod \"0f571c9e-8773-4619-a170-a3415e2d3095\" (UID: \"0f571c9e-8773-4619-a170-a3415e2d3095\") " Nov 02 10:16:01.272161 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.272126 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4028289-efd5-4100-8594-85c937a4bdc1-plugins-config-volume" pod "d4028289-efd5-4100-8594-85c937a4bdc1" (UID: "d4028289-efd5-4100-8594-85c937a4bdc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:16:01.272312 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.272215 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f571c9e-8773-4619-a170-a3415e2d3095-plugins-config-volume" pod "0f571c9e-8773-4619-a170-a3415e2d3095" (UID: "0f571c9e-8773-4619-a170-a3415e2d3095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:16:01.274047 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.274016 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4028289-efd5-4100-8594-85c937a4bdc1-kube-api-access-5kx76" pod "d4028289-efd5-4100-8594-85c937a4bdc1" (UID: "d4028289-efd5-4100-8594-85c937a4bdc1"). InnerVolumeSpecName "kube-api-access-5kx76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:16:01.274656 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.274629 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f571c9e-8773-4619-a170-a3415e2d3095-kube-api-access-tjgsv" pod "0f571c9e-8773-4619-a170-a3415e2d3095" (UID: "0f571c9e-8773-4619-a170-a3415e2d3095"). InnerVolumeSpecName "kube-api-access-tjgsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:16:01.373781 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.373722 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgsv\" (UniqueName: \"kubernetes.io/projected/0f571c9e-8773-4619-a170-a3415e2d3095-kube-api-access-tjgsv\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:16:01.373781 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.373778 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kx76\" (UniqueName: \"kubernetes.io/projected/d4028289-efd5-4100-8594-85c937a4bdc1-kube-api-access-5kx76\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:16:01.374096 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.373795 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4028289-efd5-4100-8594-85c937a4bdc1-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:16:01.374096 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.373810 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f571c9e-8773-4619-a170-a3415e2d3095-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:16:01.855682 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.855643 4467 generic.go:356] "Generic (PLEG): container finished" podID="0f571c9e-8773-4619-a170-a3415e2d3095" containerID="2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c" exitCode=0 Nov 02 10:16:01.856547 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.856349 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" Nov 02 10:16:01.857433 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.856373 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" event={"ID":"0f571c9e-8773-4619-a170-a3415e2d3095","Type":"ContainerDied","Data":"2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c"} Nov 02 10:16:01.857747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.857577 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l" event={"ID":"0f571c9e-8773-4619-a170-a3415e2d3095","Type":"ContainerDied","Data":"50ac701a7154b6d5b21c442b3c824efc7e210c13c0130dcb49b09570b95901bd"} Nov 02 10:16:01.857747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.857628 4467 scope.go:122] "RemoveContainer" containerID="2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c" Nov 02 10:16:01.862201 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.862172 4467 generic.go:356] "Generic (PLEG): container finished" podID="d4028289-efd5-4100-8594-85c937a4bdc1" containerID="19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d" exitCode=0 Nov 02 10:16:01.862312 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.862234 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" event={"ID":"d4028289-efd5-4100-8594-85c937a4bdc1","Type":"ContainerDied","Data":"19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d"} Nov 02 10:16:01.862312 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.862277 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" event={"ID":"d4028289-efd5-4100-8594-85c937a4bdc1","Type":"ContainerDied","Data":"4b99226ae2e2be4b016b18cf205404a494057d2a66ba5912916c79a1cd7041c5"} Nov 02 10:16:01.862450 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.862366 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-76459f68c5-48rr8" Nov 02 10:16:01.871591 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.871565 4467 scope.go:122] "RemoveContainer" containerID="2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c" Nov 02 10:16:01.871997 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:16:01.871974 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c\": container with ID starting with 2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c not found: ID does not exist" containerID="2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c" Nov 02 10:16:01.872091 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.872006 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c"} err="failed to get container status \"2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c\": rpc error: code = NotFound desc = could not find container \"2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c\": container with ID starting with 2757337ae761f99e6e9bb14d5af2f703848f524d6419de8b4d17ea12929c662c not found: ID does not exist" Nov 02 10:16:01.872091 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.872031 4467 scope.go:122] "RemoveContainer" containerID="19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d" Nov 02 10:16:01.881503 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.881476 4467 scope.go:122] "RemoveContainer" containerID="19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d" Nov 02 10:16:01.881808 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:16:01.881781 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d\": container with ID starting with 19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d not found: ID does not exist" containerID="19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d" Nov 02 10:16:01.881879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.881810 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d"} err="failed to get container status \"19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d\": rpc error: code = NotFound desc = could not find container \"19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d\": container with ID starting with 19c763fc8c6671075a2f65249c82adfff9be0aab7cbcc1f50d44f5b2faa8625d not found: ID does not exist" Nov 02 10:16:01.889582 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.889545 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l"] Nov 02 10:16:01.910856 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.910818 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-6c45b49d64-2dd4l"] Nov 02 10:16:01.923908 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.923849 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-76459f68c5-48rr8"] Nov 02 10:16:01.935083 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:01.935045 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-76459f68c5-48rr8"] Nov 02 10:16:03.359645 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:03.359600 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f571c9e-8773-4619-a170-a3415e2d3095" path="/var/lib/kubelet/pods/0f571c9e-8773-4619-a170-a3415e2d3095/volumes" Nov 02 10:16:03.360826 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:03.360799 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4028289-efd5-4100-8594-85c937a4bdc1" path="/var/lib/kubelet/pods/d4028289-efd5-4100-8594-85c937a4bdc1/volumes" Nov 02 10:16:03.606939 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:03.606011 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:16:03.606939 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:03.606083 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:16:10.836724 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:10.836688 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:16:12.638784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:12.638745 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:16:17.364813 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:17.364768 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:16:26.854724 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:16:26.854682 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:16:28.404600 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:28.404568 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:16:30.761254 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:30.761220 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:16:48.170694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:16:48.170642 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:16:56.854571 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:16:56.854533 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:17:12.704820 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:12.704785 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:17:26.853780 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:17:26.853729 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:17:31.540627 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:31.540585 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9"] Nov 02 10:17:31.684315 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:31.684284 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:17:31.871357 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:31.871321 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:17:32.058033 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:32.057999 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:17:33.176970 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:33.176929 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:17:35.691913 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:35.691027 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:17:35.691913 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:35.691149 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:17:36.541192 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.541128 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" podUID="d3c26b21-7d51-4d0e-859a-bb4a7844881e" containerName="payload-processing" containerID="cri-o://07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab" gracePeriod=25 Nov 02 10:17:36.814154 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.814123 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:17:36.976747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.976698 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-provider-credentials\") pod \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " Nov 02 10:17:36.976978 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.976757 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-kube-api-access-4tx5l\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-kube-api-access-4tx5l\") pod \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " Nov 02 10:17:36.976978 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.976855 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-routing-overlay-volume\") pod \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " Nov 02 10:17:36.976978 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.976935 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-plugins-config-volume\") pod \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\" (UID: \"d3c26b21-7d51-4d0e-859a-bb4a7844881e\") " Nov 02 10:17:36.977271 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.977231 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-routing-overlay-volume" pod "d3c26b21-7d51-4d0e-859a-bb4a7844881e" (UID: "d3c26b21-7d51-4d0e-859a-bb4a7844881e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:17:36.977512 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.977487 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-plugins-config-volume" pod "d3c26b21-7d51-4d0e-859a-bb4a7844881e" (UID: "d3c26b21-7d51-4d0e-859a-bb4a7844881e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:17:36.979376 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.979344 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-provider-credentials" pod "d3c26b21-7d51-4d0e-859a-bb4a7844881e" (UID: "d3c26b21-7d51-4d0e-859a-bb4a7844881e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:17:36.979612 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:36.979581 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-kube-api-access-4tx5l" pod "d3c26b21-7d51-4d0e-859a-bb4a7844881e" (UID: "d3c26b21-7d51-4d0e-859a-bb4a7844881e"). InnerVolumeSpecName "kube-api-access-4tx5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:17:37.078592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.078542 4467 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-provider-credentials\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:17:37.078592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.078586 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tx5l\" (UniqueName: \"kubernetes.io/projected/d3c26b21-7d51-4d0e-859a-bb4a7844881e-kube-api-access-4tx5l\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:17:37.078592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.078602 4467 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-routing-overlay-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:17:37.078976 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.078619 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c26b21-7d51-4d0e-859a-bb4a7844881e-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:17:37.369481 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.369441 4467 generic.go:356] "Generic (PLEG): container finished" podID="d3c26b21-7d51-4d0e-859a-bb4a7844881e" containerID="07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab" exitCode=0 Nov 02 10:17:37.369689 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.369523 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" event={"ID":"d3c26b21-7d51-4d0e-859a-bb4a7844881e","Type":"ContainerDied","Data":"07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab"} Nov 02 10:17:37.369689 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.369565 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" Nov 02 10:17:37.369689 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.369593 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9" event={"ID":"d3c26b21-7d51-4d0e-859a-bb4a7844881e","Type":"ContainerDied","Data":"c6c8a7a4cdede9cab2882c476bd803d57d943df0812405c0f1b7f5644793b67f"} Nov 02 10:17:37.369689 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.369623 4467 scope.go:122] "RemoveContainer" containerID="07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab" Nov 02 10:17:37.386658 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.386637 4467 scope.go:122] "RemoveContainer" containerID="07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab" Nov 02 10:17:37.387147 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:17:37.387113 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab\": container with ID starting with 07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab not found: ID does not exist" containerID="07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab" Nov 02 10:17:37.387294 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.387157 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab"} err="failed to get container status \"07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab\": rpc error: code = NotFound desc = could not find container \"07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab\": container with ID starting with 07d8447c6a6e9556059c778796a4601345f92ca96737df1e22f7ced3edc92eab not found: ID does not exist" Nov 02 10:17:37.400023 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.399977 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9"] Nov 02 10:17:37.406002 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:37.405970 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-kt9n9"] Nov 02 10:17:38.663930 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:38.663055 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:17:39.351914 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:39.350936 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c26b21-7d51-4d0e-859a-bb4a7844881e" path="/var/lib/kubelet/pods/d3c26b21-7d51-4d0e-859a-bb4a7844881e/volumes" Nov 02 10:17:48.687247 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:48.687209 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:17:53.266875 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:17:53.266835 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:17:56.854261 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:17:56.854226 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:18:06.294221 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:18:06.294127 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:18:12.744502 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:18:12.744464 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:18:19.050179 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:18:19.050144 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:18:26.146130 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:18:26.146089 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:18:26.854283 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:18:26.854242 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:18:43.902589 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:18:43.902547 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:18:54.159277 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:18:54.159240 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:18:56.854734 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:18:56.854699 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:18:59.150916 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:18:59.150028 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:19:12.826485 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:12.826451 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:19:14.960348 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:14.960313 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:19:18.808311 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:18.808273 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:19:20.687062 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:20.687019 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:19:20.687726 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:20.687671 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:19:26.853813 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:19:26.853774 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:19:32.273273 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:32.273234 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:19:37.336926 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:37.336255 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:19:40.960391 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:40.960351 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:19:47.555250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:47.555187 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:19:47.739089 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:47.739054 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:19:48.759222 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:48.759186 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:19:50.350427 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:50.350374 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:19:51.707915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:51.707016 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:19:51.707915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:19:51.707105 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:19:56.853738 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:19:56.853702 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:20:02.677826 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:02.677787 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:20:12.849025 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:12.848987 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:20:13.382526 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:13.382493 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:20:18.057680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:18.057641 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:20:20.678047 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:20.678003 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7prvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:20:20.678462 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:20.678065 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:20:26.853802 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:20:26.853759 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:20:29.480953 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:29.480918 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:20:33.366219 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:33.366182 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:20:36.265456 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:36.265421 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:20:52.553829 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:20:52.553794 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:20:56.853052 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:20:56.853017 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:21:02.794205 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:02.794168 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:21:12.815048 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:12.815003 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:19.593714 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:19.593673 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:21:21.703914 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:21.703004 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:21:21.703914 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:21.703099 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:21:23.270305 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:23.270264 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:23.791497 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:23.791460 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:23.976952 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:23.976921 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:24.572278 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:24.572239 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:21:26.856650 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:21:26.856616 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:21:26.902688 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:26.902654 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:27.086775 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.086733 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:27.991226 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.991171 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc"] Nov 02 10:21:27.992354 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.991874 4467 cpu_manager.go:414] "containerMap: removing container" podUID="0f571c9e-8773-4619-a170-a3415e2d3095" containerName="payload-pre-processing" Nov 02 10:21:27.992553 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.992532 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f571c9e-8773-4619-a170-a3415e2d3095" containerName="payload-pre-processing" Nov 02 10:21:27.992709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.992698 4467 cpu_manager.go:414] "containerMap: removing container" podUID="d3c26b21-7d51-4d0e-859a-bb4a7844881e" containerName="payload-processing" Nov 02 10:21:27.992856 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.992842 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c26b21-7d51-4d0e-859a-bb4a7844881e" containerName="payload-processing" Nov 02 10:21:27.992993 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.992981 4467 cpu_manager.go:414] "containerMap: removing container" podUID="d4028289-efd5-4100-8594-85c937a4bdc1" containerName="payload-processing" Nov 02 10:21:27.993074 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.993064 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4028289-efd5-4100-8594-85c937a4bdc1" containerName="payload-processing" Nov 02 10:21:27.993318 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.993302 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c26b21-7d51-4d0e-859a-bb4a7844881e" containerName="payload-processing" Nov 02 10:21:27.993416 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.993405 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4028289-efd5-4100-8594-85c937a4bdc1" containerName="payload-processing" Nov 02 10:21:27.993598 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.993547 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f571c9e-8773-4619-a170-a3415e2d3095" containerName="payload-pre-processing" Nov 02 10:21:27.997245 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:27.997224 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.010515 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.010487 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc"] Nov 02 10:21:28.011100 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.011078 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-8d45bb-serving-cert\"" Nov 02 10:21:28.129788 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.129753 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw"] Nov 02 10:21:28.133635 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.133601 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.138684 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.138642 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.138947 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.138925 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.139047 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.139006 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.139107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.139043 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqz6q\" (UniqueName: \"kubernetes.io/projected/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-kube-api-access-zqz6q\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.141320 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.141292 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-2fa7c6ca-dockercfg-zf4nx\"" Nov 02 10:21:28.143236 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.143212 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8d45bb\"" Nov 02 10:21:28.143744 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.143720 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw"] Nov 02 10:21:28.167481 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.167439 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:28.168469 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.168446 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf"] Nov 02 10:21:28.172651 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.172625 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:28.181415 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.181385 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf"] Nov 02 10:21:28.241016 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.240804 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.241370 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.241307 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.241521 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.241506 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.241709 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.241631 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqz6q\" (UniqueName: \"kubernetes.io/projected/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-kube-api-access-zqz6q\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.241827 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.241794 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xzcm\" (UniqueName: \"kubernetes.io/projected/b550974a-8184-4ef6-b9cc-4379fbfd6b97-kube-api-access-2xzcm\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw\" (UID: \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.241928 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.241836 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsg49\" (UniqueName: \"kubernetes.io/projected/a08ebc8d-9596-4f0e-8696-564800a2ba85-kube-api-access-nsg49\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf\" (UID: \"a08ebc8d-9596-4f0e-8696-564800a2ba85\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:28.241928 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.241913 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ebc8d-9596-4f0e-8696-564800a2ba85-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf\" (UID: \"a08ebc8d-9596-4f0e-8696-564800a2ba85\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:28.242054 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.241966 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b550974a-8184-4ef6-b9cc-4379fbfd6b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw\" (UID: \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.243112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.243081 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.245262 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.245237 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.245866 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.245841 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.255025 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.254962 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqz6q\" (UniqueName: \"kubernetes.io/projected/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-kube-api-access-zqz6q\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.320070 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.320035 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:28.343009 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.342978 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b550974a-8184-4ef6-b9cc-4379fbfd6b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw\" (UID: \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.343204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.343111 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xzcm\" (UniqueName: \"kubernetes.io/projected/b550974a-8184-4ef6-b9cc-4379fbfd6b97-kube-api-access-2xzcm\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw\" (UID: \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.343204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.343151 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsg49\" (UniqueName: \"kubernetes.io/projected/a08ebc8d-9596-4f0e-8696-564800a2ba85-kube-api-access-nsg49\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf\" (UID: \"a08ebc8d-9596-4f0e-8696-564800a2ba85\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:28.343333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.343220 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ebc8d-9596-4f0e-8696-564800a2ba85-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf\" (UID: \"a08ebc8d-9596-4f0e-8696-564800a2ba85\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:28.343979 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.343950 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b550974a-8184-4ef6-b9cc-4379fbfd6b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw\" (UID: \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.344090 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.343951 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ebc8d-9596-4f0e-8696-564800a2ba85-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf\" (UID: \"a08ebc8d-9596-4f0e-8696-564800a2ba85\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:28.356395 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.356358 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsg49\" (UniqueName: \"kubernetes.io/projected/a08ebc8d-9596-4f0e-8696-564800a2ba85-kube-api-access-nsg49\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf\" (UID: \"a08ebc8d-9596-4f0e-8696-564800a2ba85\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:28.357457 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.357425 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xzcm\" (UniqueName: \"kubernetes.io/projected/b550974a-8184-4ef6-b9cc-4379fbfd6b97-kube-api-access-2xzcm\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw\" (UID: \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.397787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.397753 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:28.454091 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.454053 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.490376 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.490338 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:28.609381 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.609348 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc"] Nov 02 10:21:28.610926 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:21:28.610873 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb37a34_3bf0_4ff1_84db_6d6b8c8b0af7.slice/crio-e6552af41e737acc708f0fd62b0a5a137effc35da7244738a7040db0426637e9 WatchSource:0}: Error finding container e6552af41e737acc708f0fd62b0a5a137effc35da7244738a7040db0426637e9: Status 404 returned error can't find the container with id e6552af41e737acc708f0fd62b0a5a137effc35da7244738a7040db0426637e9 Nov 02 10:21:28.612271 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.612252 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 10:21:28.745499 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.745357 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw"] Nov 02 10:21:28.748288 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:21:28.748259 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb550974a_8184_4ef6_b9cc_4379fbfd6b97.slice/crio-66d413d1426466863955d16e7cdd72a4133f76263dec50fdef733adfdab2da3f WatchSource:0}: Error finding container 66d413d1426466863955d16e7cdd72a4133f76263dec50fdef733adfdab2da3f: Status 404 returned error can't find the container with id 66d413d1426466863955d16e7cdd72a4133f76263dec50fdef733adfdab2da3f Nov 02 10:21:28.792802 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.792773 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf"] Nov 02 10:21:28.796409 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:21:28.796370 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08ebc8d_9596_4f0e_8696_564800a2ba85.slice/crio-ca490c7f055a8c4d0434f7cdb031181017e1306173a9fef2338e84a83923e4d0 WatchSource:0}: Error finding container ca490c7f055a8c4d0434f7cdb031181017e1306173a9fef2338e84a83923e4d0: Status 404 returned error can't find the container with id ca490c7f055a8c4d0434f7cdb031181017e1306173a9fef2338e84a83923e4d0 Nov 02 10:21:28.858577 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.858547 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" event={"ID":"a08ebc8d-9596-4f0e-8696-564800a2ba85","Type":"ContainerStarted","Data":"ca490c7f055a8c4d0434f7cdb031181017e1306173a9fef2338e84a83923e4d0"} Nov 02 10:21:28.860351 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.860316 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" event={"ID":"b550974a-8184-4ef6-b9cc-4379fbfd6b97","Type":"ContainerStarted","Data":"cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba"} Nov 02 10:21:28.860478 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.860371 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" event={"ID":"b550974a-8184-4ef6-b9cc-4379fbfd6b97","Type":"ContainerStarted","Data":"66d413d1426466863955d16e7cdd72a4133f76263dec50fdef733adfdab2da3f"} Nov 02 10:21:28.860478 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.860415 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:28.861633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.861611 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" event={"ID":"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7","Type":"ContainerStarted","Data":"e6552af41e737acc708f0fd62b0a5a137effc35da7244738a7040db0426637e9"} Nov 02 10:21:28.887724 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:28.887660 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" podStartSLOduration=0.887639791 podStartE2EDuration="887.639791ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:21:28.880463335 +0000 UTC m=+2767.720405331" watchObservedRunningTime="2026-11-02 10:21:28.887639791 +0000 UTC m=+2767.727581787" Nov 02 10:21:29.124861 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:29.124834 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:29.816591 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:29.816562 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:29.868340 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:29.868293 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" event={"ID":"a08ebc8d-9596-4f0e-8696-564800a2ba85","Type":"ContainerStarted","Data":"b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7"} Nov 02 10:21:29.868544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:29.868372 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:29.870122 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:29.870092 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" event={"ID":"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7","Type":"ContainerStarted","Data":"bd3e6b823575dd0ee607c0ac6a16d7376d888cd0189ab75d97d478f948ba78dc"} Nov 02 10:21:29.870230 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:29.870158 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:29.887325 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:29.887273 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" podStartSLOduration=1.887257939 podStartE2EDuration="1.887257939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:21:29.886767614 +0000 UTC m=+2768.726709610" watchObservedRunningTime="2026-11-02 10:21:29.887257939 +0000 UTC m=+2768.727199937" Nov 02 10:21:29.924138 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:29.924094 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" podStartSLOduration=2.451539207 podStartE2EDuration="2.92406823s" totalImagesPullingTime="472.529023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:21:29.913283831 +0000 UTC m=+2768.753225825" watchObservedRunningTime="2026-11-02 10:21:29.92406823 +0000 UTC m=+2768.764010225" Nov 02 10:21:30.612741 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:30.612701 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:30.678666 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:30.678634 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:21:31.513948 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:31.513874 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:32.278618 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:32.278573 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:32.924791 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:32.924752 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:33.611238 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:33.611206 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:34.672422 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:34.672389 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:35.050333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:35.050255 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:35.797715 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:35.797683 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:35.881914 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:35.881863 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:21:36.647792 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:36.647762 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:37.507604 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:37.507531 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:38.226718 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:38.226682 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:39.007796 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:39.007763 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:39.709090 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:39.709059 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:39.873687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:39.873655 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:21:40.408060 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:40.408025 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:40.878587 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:40.878557 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:21:41.221730 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:41.221642 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:41.917107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:41.917078 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:42.122090 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:42.122054 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:42.543598 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:42.543570 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:42.737215 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:42.737181 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:43.614877 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:43.614844 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:43.934445 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:43.934363 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:44.430300 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:44.430269 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:45.121496 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:45.121457 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:45.584940 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:45.584880 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:21:45.975824 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:45.975749 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:46.721756 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:46.721718 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:47.520841 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:47.520796 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:48.296360 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:48.296309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:49.000364 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:49.000331 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:49.804383 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:49.804314 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:50.503266 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:50.503228 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:51.209150 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:51.209103 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:51.921273 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:51.921223 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:52.610248 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:52.610209 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:53.305848 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:53.305807 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:54.006400 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:54.006367 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:54.710272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:54.710234 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:55.409173 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:55.409140 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:56.208353 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:56.208321 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:56.854269 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:21:56.854234 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:21:56.923097 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:56.923061 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:57.603192 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:57.603147 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:58.303017 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:58.302983 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:58.793632 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:58.793596 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:59.011089 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:59.011058 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:21:59.716264 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:21:59.716227 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:00.424455 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:00.424421 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:01.229057 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:01.229001 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:02.025120 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:02.025087 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:02.493300 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:02.493261 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:22:02.868154 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:02.868118 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:03.701870 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:03.701822 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:04.407998 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:04.407965 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:05.148228 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:05.148196 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:05.823596 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:05.823557 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:06.511512 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:06.511475 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:07.213766 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:07.213734 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:07.907186 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:07.907150 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:08.798163 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:08.798132 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:09.506730 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:09.506689 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:10.210699 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:10.210670 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:10.906456 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:10.906411 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:11.611017 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:11.610979 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:12.329263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:12.329229 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:12.947142 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:12.947111 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:13.221396 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:13.221312 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:14.032523 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:14.032491 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:14.726221 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:14.726185 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:15.499650 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:15.499614 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:16.197848 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:16.197813 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:17.104592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:17.104559 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:17.825456 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:17.825425 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:18.505868 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:18.505831 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:19.209120 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:19.209084 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:19.908759 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:19.908722 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:20.617828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:20.617799 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:21.427435 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:21.427401 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:22.148912 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:22.148854 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:22.895986 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:22.895950 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:23.707832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:23.707787 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:24.525929 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:24.518820 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:24.871674 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:24.871640 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:25.320455 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:25.320408 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:25.496069 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:25.496039 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:25.780268 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:25.780182 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:22:26.023762 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:26.023727 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:26.821633 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:26.821605 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:26.854335 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:22:26.854290 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:22:27.522194 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:27.522163 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:28.211261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:28.211229 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:28.928071 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:28.928037 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:29.818623 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:29.818584 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:29.989444 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:29.989411 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:30.508966 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:30.508925 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:30.687188 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:30.687154 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:31.416543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:31.416509 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:31.707047 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:31.706955 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf"] Nov 02 10:22:31.727474 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:31.727442 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw"] Nov 02 10:22:31.874509 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:31.874477 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:32.057476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:32.057448 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:32.247385 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:32.247357 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:32.423178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:32.423093 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:32.982912 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:32.982850 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc"] Nov 02 10:22:32.983167 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:32.983141 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" podUID="cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" containerName="maas-api" containerID="cri-o://bd3e6b823575dd0ee607c0ac6a16d7376d888cd0189ab75d97d478f948ba78dc" gracePeriod=30 Nov 02 10:22:33.119282 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.119236 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:33.210434 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.210398 4467 generic.go:356] "Generic (PLEG): container finished" podID="cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" containerID="bd3e6b823575dd0ee607c0ac6a16d7376d888cd0189ab75d97d478f948ba78dc" exitCode=0 Nov 02 10:22:33.210612 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.210492 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" event={"ID":"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7","Type":"ContainerDied","Data":"bd3e6b823575dd0ee607c0ac6a16d7376d888cd0189ab75d97d478f948ba78dc"} Nov 02 10:22:33.249263 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.249203 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:22:33.290575 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.290543 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-metrics-certs\") pod \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " Nov 02 10:22:33.290765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.290613 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-maas-api-tls\") pod \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " Nov 02 10:22:33.290765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.290689 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-openshift-service-ca\") pod \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " Nov 02 10:22:33.290765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.290717 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-kube-api-access-zqz6q\" (UniqueName: \"kubernetes.io/projected/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-kube-api-access-zqz6q\") pod \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\" (UID: \"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7\") " Nov 02 10:22:33.291131 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.291102 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-openshift-service-ca" pod "cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" (UID: "cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:22:33.293274 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.293247 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-kube-api-access-zqz6q" pod "cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" (UID: "cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7"). InnerVolumeSpecName "kube-api-access-zqz6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:22:33.293784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.293751 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-metrics-certs" pod "cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" (UID: "cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:22:33.293918 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.293871 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-maas-api-tls" pod "cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" (UID: "cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:22:33.391544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.391516 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:22:33.391544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.391542 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:22:33.391544 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.391552 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqz6q\" (UniqueName: \"kubernetes.io/projected/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-kube-api-access-zqz6q\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:22:33.391799 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:33.391563 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:22:34.216965 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:34.216930 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" event={"ID":"cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7","Type":"ContainerDied","Data":"e6552af41e737acc708f0fd62b0a5a137effc35da7244738a7040db0426637e9"} Nov 02 10:22:34.216965 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:34.216953 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc" Nov 02 10:22:34.217491 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:34.217014 4467 scope.go:122] "RemoveContainer" containerID="bd3e6b823575dd0ee607c0ac6a16d7376d888cd0189ab75d97d478f948ba78dc" Nov 02 10:22:34.236241 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:34.236216 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc"] Nov 02 10:22:34.240532 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:34.240509 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-7s5pc"] Nov 02 10:22:34.976343 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:34.976310 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:22:35.334399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:35.334361 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" path="/var/lib/kubelet/pods/cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7/volumes" Nov 02 10:22:36.707828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:36.707783 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" podUID="a08ebc8d-9596-4f0e-8696-564800a2ba85" containerName="payload-processing" containerID="cri-o://b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7" gracePeriod=25 Nov 02 10:22:36.728503 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:36.728470 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" podUID="b550974a-8184-4ef6-b9cc-4379fbfd6b97" containerName="payload-pre-processing" containerID="cri-o://cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba" gracePeriod=25 Nov 02 10:22:38.169124 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:38.169093 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:38.351935 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:38.351872 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:39.871636 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:39.871597 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" podUID="b550974a-8184-4ef6-b9cc-4379fbfd6b97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:22:40.877327 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:40.877284 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" podUID="a08ebc8d-9596-4f0e-8696-564800a2ba85" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:22:49.871993 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:49.871951 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" podUID="b550974a-8184-4ef6-b9cc-4379fbfd6b97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:22:50.876974 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:50.876936 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" podUID="a08ebc8d-9596-4f0e-8696-564800a2ba85" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 10:22:54.437022 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:54.436980 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:22:56.858346 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:22:56.858305 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:22:56.976945 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:56.976877 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:22:57.006902 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.006860 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:22:57.128007 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.127975 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08ebc8d-9596-4f0e-8696-564800a2ba85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ebc8d-9596-4f0e-8696-564800a2ba85-plugins-config-volume\") pod \"a08ebc8d-9596-4f0e-8696-564800a2ba85\" (UID: \"a08ebc8d-9596-4f0e-8696-564800a2ba85\") " Nov 02 10:22:57.128178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.128029 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08ebc8d-9596-4f0e-8696-564800a2ba85-kube-api-access-nsg49\" (UniqueName: \"kubernetes.io/projected/a08ebc8d-9596-4f0e-8696-564800a2ba85-kube-api-access-nsg49\") pod \"a08ebc8d-9596-4f0e-8696-564800a2ba85\" (UID: \"a08ebc8d-9596-4f0e-8696-564800a2ba85\") " Nov 02 10:22:57.128178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.128069 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b550974a-8184-4ef6-b9cc-4379fbfd6b97-kube-api-access-2xzcm\" (UniqueName: \"kubernetes.io/projected/b550974a-8184-4ef6-b9cc-4379fbfd6b97-kube-api-access-2xzcm\") pod \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\" (UID: \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\") " Nov 02 10:22:57.128178 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.128115 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b550974a-8184-4ef6-b9cc-4379fbfd6b97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b550974a-8184-4ef6-b9cc-4379fbfd6b97-plugins-config-volume\") pod \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\" (UID: \"b550974a-8184-4ef6-b9cc-4379fbfd6b97\") " Nov 02 10:22:57.128462 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.128436 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08ebc8d-9596-4f0e-8696-564800a2ba85-plugins-config-volume" pod "a08ebc8d-9596-4f0e-8696-564800a2ba85" (UID: "a08ebc8d-9596-4f0e-8696-564800a2ba85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:22:57.128523 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.128509 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b550974a-8184-4ef6-b9cc-4379fbfd6b97-plugins-config-volume" pod "b550974a-8184-4ef6-b9cc-4379fbfd6b97" (UID: "b550974a-8184-4ef6-b9cc-4379fbfd6b97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:22:57.130257 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.130231 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b550974a-8184-4ef6-b9cc-4379fbfd6b97-kube-api-access-2xzcm" pod "b550974a-8184-4ef6-b9cc-4379fbfd6b97" (UID: "b550974a-8184-4ef6-b9cc-4379fbfd6b97"). InnerVolumeSpecName "kube-api-access-2xzcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:22:57.130257 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.130246 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08ebc8d-9596-4f0e-8696-564800a2ba85-kube-api-access-nsg49" pod "a08ebc8d-9596-4f0e-8696-564800a2ba85" (UID: "a08ebc8d-9596-4f0e-8696-564800a2ba85"). InnerVolumeSpecName "kube-api-access-nsg49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:22:57.229718 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.229621 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xzcm\" (UniqueName: \"kubernetes.io/projected/b550974a-8184-4ef6-b9cc-4379fbfd6b97-kube-api-access-2xzcm\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:22:57.229718 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.229663 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b550974a-8184-4ef6-b9cc-4379fbfd6b97-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:22:57.229718 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.229672 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ebc8d-9596-4f0e-8696-564800a2ba85-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:22:57.229718 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.229683 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsg49\" (UniqueName: \"kubernetes.io/projected/a08ebc8d-9596-4f0e-8696-564800a2ba85-kube-api-access-nsg49\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:22:57.335076 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.335043 4467 generic.go:356] "Generic (PLEG): container finished" podID="a08ebc8d-9596-4f0e-8696-564800a2ba85" containerID="b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7" exitCode=0 Nov 02 10:22:57.335261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.335104 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" Nov 02 10:22:57.335261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.335130 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" event={"ID":"a08ebc8d-9596-4f0e-8696-564800a2ba85","Type":"ContainerDied","Data":"b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7"} Nov 02 10:22:57.335261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.335190 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf" event={"ID":"a08ebc8d-9596-4f0e-8696-564800a2ba85","Type":"ContainerDied","Data":"ca490c7f055a8c4d0434f7cdb031181017e1306173a9fef2338e84a83923e4d0"} Nov 02 10:22:57.335261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.335220 4467 scope.go:122] "RemoveContainer" containerID="b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7" Nov 02 10:22:57.336511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.336489 4467 generic.go:356] "Generic (PLEG): container finished" podID="b550974a-8184-4ef6-b9cc-4379fbfd6b97" containerID="cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba" exitCode=0 Nov 02 10:22:57.336640 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.336533 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" Nov 02 10:22:57.336640 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.336558 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" event={"ID":"b550974a-8184-4ef6-b9cc-4379fbfd6b97","Type":"ContainerDied","Data":"cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba"} Nov 02 10:22:57.336640 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.336596 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw" event={"ID":"b550974a-8184-4ef6-b9cc-4379fbfd6b97","Type":"ContainerDied","Data":"66d413d1426466863955d16e7cdd72a4133f76263dec50fdef733adfdab2da3f"} Nov 02 10:22:57.344404 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.344382 4467 scope.go:122] "RemoveContainer" containerID="b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7" Nov 02 10:22:57.344712 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:22:57.344684 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7\": container with ID starting with b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7 not found: ID does not exist" containerID="b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7" Nov 02 10:22:57.344795 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.344724 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7"} err="failed to get container status \"b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7\": rpc error: code = NotFound desc = could not find container \"b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7\": container with ID starting with b168954d1b2a51fc05d4bb13b2df4cbc2206fd62783d0fb7bb7c1f8b3519c9f7 not found: ID does not exist" Nov 02 10:22:57.344795 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.344747 4467 scope.go:122] "RemoveContainer" containerID="cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba" Nov 02 10:22:57.355159 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.355143 4467 scope.go:122] "RemoveContainer" containerID="cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba" Nov 02 10:22:57.355410 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:22:57.355388 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba\": container with ID starting with cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba not found: ID does not exist" containerID="cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba" Nov 02 10:22:57.355463 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.355420 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba"} err="failed to get container status \"cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba\": rpc error: code = NotFound desc = could not find container \"cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba\": container with ID starting with cb3e1256a855ad218df24f68d1320d04424013e37b75c764aea85a121b30c8ba not found: ID does not exist" Nov 02 10:22:57.369321 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.369294 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw"] Nov 02 10:22:57.375222 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.375197 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd776wgbw"] Nov 02 10:22:57.387753 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.387726 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf"] Nov 02 10:22:57.392193 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:57.392169 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6b2gvwf"] Nov 02 10:22:59.330317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:59.330283 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08ebc8d-9596-4f0e-8696-564800a2ba85" path="/var/lib/kubelet/pods/a08ebc8d-9596-4f0e-8696-564800a2ba85/volumes" Nov 02 10:22:59.330730 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:22:59.330621 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b550974a-8184-4ef6-b9cc-4379fbfd6b97" path="/var/lib/kubelet/pods/b550974a-8184-4ef6-b9cc-4379fbfd6b97/volumes" Nov 02 10:23:03.335104 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:03.335071 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:03.515820 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:03.515779 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:04.369010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.368972 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt"] Nov 02 10:23:04.369705 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369682 4467 cpu_manager.go:414] "containerMap: removing container" podUID="cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" containerName="maas-api" Nov 02 10:23:04.369769 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369709 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" containerName="maas-api" Nov 02 10:23:04.369769 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369751 4467 cpu_manager.go:414] "containerMap: removing container" podUID="b550974a-8184-4ef6-b9cc-4379fbfd6b97" containerName="payload-pre-processing" Nov 02 10:23:04.369769 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369760 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b550974a-8184-4ef6-b9cc-4379fbfd6b97" containerName="payload-pre-processing" Nov 02 10:23:04.369870 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369788 4467 cpu_manager.go:414] "containerMap: removing container" podUID="a08ebc8d-9596-4f0e-8696-564800a2ba85" containerName="payload-processing" Nov 02 10:23:04.369870 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369796 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08ebc8d-9596-4f0e-8696-564800a2ba85" containerName="payload-processing" Nov 02 10:23:04.369958 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369942 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="b550974a-8184-4ef6-b9cc-4379fbfd6b97" containerName="payload-pre-processing" Nov 02 10:23:04.369999 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369965 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08ebc8d-9596-4f0e-8696-564800a2ba85" containerName="payload-processing" Nov 02 10:23:04.369999 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.369979 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb37a34-3bf0-4ff1-84db-6d6b8c8b0af7" containerName="maas-api" Nov 02 10:23:04.375241 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.375220 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.388635 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.388609 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-8d45bb-serving-cert\"" Nov 02 10:23:04.391601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.391565 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt"] Nov 02 10:23:04.391962 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.391936 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.392052 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.391985 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.392117 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.392095 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.392278 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.392260 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstsv\" (UniqueName: \"kubernetes.io/projected/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-kube-api-access-wstsv\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.477651 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.477618 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k"] Nov 02 10:23:04.481839 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.481812 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6"] Nov 02 10:23:04.482036 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.481948 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:04.485527 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.485503 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:04.488511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.488490 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8d45bb\"" Nov 02 10:23:04.493278 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.493255 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-2fa7c6ca-dockercfg-nnkd8\"" Nov 02 10:23:04.493424 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.493346 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.493424 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.493386 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.493592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.493424 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.493592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.493528 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstsv\" (UniqueName: \"kubernetes.io/projected/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-kube-api-access-wstsv\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.500927 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.495325 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k"] Nov 02 10:23:04.500927 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:23:04.495438 4467 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-serving-cert: secret "maas-api-e2e-worker-single-main-8d45bb-serving-cert" not found Nov 02 10:23:04.500927 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:23:04.495507 4467 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls podName:71409ee6-b66e-4dcc-90f4-ebf2e0e53187 nodeName:}" failed. No retries permitted until 2026-11-02 10:23:04.995487743 +0000 UTC m=+2863.835429729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls") pod "maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" (UID: "71409ee6-b66e-4dcc-90f4-ebf2e0e53187") : secret "maas-api-e2e-worker-single-main-8d45bb-serving-cert" not found Nov 02 10:23:04.500927 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.496103 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.500927 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.497238 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.500927 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.500657 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6"] Nov 02 10:23:04.515591 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.515530 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstsv\" (UniqueName: \"kubernetes.io/projected/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-kube-api-access-wstsv\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:04.517204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.517176 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:04.594444 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.594413 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80631528-0928-4785-8cff-132d21aa9fbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6\" (UID: \"80631528-0928-4785-8cff-132d21aa9fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:04.594607 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.594462 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8cgl\" (UniqueName: \"kubernetes.io/projected/d099db5a-b4e0-47d9-8840-52d37187c172-kube-api-access-x8cgl\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k\" (UID: \"d099db5a-b4e0-47d9-8840-52d37187c172\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:04.594607 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.594536 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d099db5a-b4e0-47d9-8840-52d37187c172-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k\" (UID: \"d099db5a-b4e0-47d9-8840-52d37187c172\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:04.594764 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.594734 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgq9\" (UniqueName: \"kubernetes.io/projected/80631528-0928-4785-8cff-132d21aa9fbe-kube-api-access-jrgq9\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6\" (UID: \"80631528-0928-4785-8cff-132d21aa9fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:04.695648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.695559 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgq9\" (UniqueName: \"kubernetes.io/projected/80631528-0928-4785-8cff-132d21aa9fbe-kube-api-access-jrgq9\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6\" (UID: \"80631528-0928-4785-8cff-132d21aa9fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:04.695648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.695625 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80631528-0928-4785-8cff-132d21aa9fbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6\" (UID: \"80631528-0928-4785-8cff-132d21aa9fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:04.695916 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.695660 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8cgl\" (UniqueName: \"kubernetes.io/projected/d099db5a-b4e0-47d9-8840-52d37187c172-kube-api-access-x8cgl\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k\" (UID: \"d099db5a-b4e0-47d9-8840-52d37187c172\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:04.695916 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.695686 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d099db5a-b4e0-47d9-8840-52d37187c172-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k\" (UID: \"d099db5a-b4e0-47d9-8840-52d37187c172\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:04.696450 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.696428 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d099db5a-b4e0-47d9-8840-52d37187c172-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k\" (UID: \"d099db5a-b4e0-47d9-8840-52d37187c172\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:04.696497 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.696449 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80631528-0928-4785-8cff-132d21aa9fbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6\" (UID: \"80631528-0928-4785-8cff-132d21aa9fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:04.698216 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.698193 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:04.708380 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.708358 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgq9\" (UniqueName: \"kubernetes.io/projected/80631528-0928-4785-8cff-132d21aa9fbe-kube-api-access-jrgq9\") pod \"payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6\" (UID: \"80631528-0928-4785-8cff-132d21aa9fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:04.708380 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.708377 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8cgl\" (UniqueName: \"kubernetes.io/projected/d099db5a-b4e0-47d9-8840-52d37187c172-kube-api-access-x8cgl\") pod \"payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k\" (UID: \"d099db5a-b4e0-47d9-8840-52d37187c172\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:04.804714 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.804679 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:04.832262 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.832235 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:04.882355 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.882321 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:04.998647 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:04.998556 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:05.000998 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.000974 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:05.152381 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.152355 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k"] Nov 02 10:23:05.154414 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:23:05.154387 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80631528_0928_4785_8cff_132d21aa9fbe.slice/crio-8c60cb3bc9b20a50627d81741220f38a3e6ef1408ed999959c3fa38a14373648 WatchSource:0}: Error finding container 8c60cb3bc9b20a50627d81741220f38a3e6ef1408ed999959c3fa38a14373648: Status 404 returned error can't find the container with id 8c60cb3bc9b20a50627d81741220f38a3e6ef1408ed999959c3fa38a14373648 Nov 02 10:23:05.154925 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:23:05.154876 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd099db5a_b4e0_47d9_8840_52d37187c172.slice/crio-d518ce6797d1e4dbdfc5cda385adb92a190b9ed87c99d262b179485798abdb2e WatchSource:0}: Error finding container d518ce6797d1e4dbdfc5cda385adb92a190b9ed87c99d262b179485798abdb2e: Status 404 returned error can't find the container with id d518ce6797d1e4dbdfc5cda385adb92a190b9ed87c99d262b179485798abdb2e Nov 02 10:23:05.155082 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.155057 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6"] Nov 02 10:23:05.299877 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.299790 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:05.386272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.386237 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" event={"ID":"80631528-0928-4785-8cff-132d21aa9fbe","Type":"ContainerStarted","Data":"5d8de7b930705593208436079bfd1e456ef139275d7614d690352f92b7a9eb03"} Nov 02 10:23:05.386736 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.386291 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" event={"ID":"80631528-0928-4785-8cff-132d21aa9fbe","Type":"ContainerStarted","Data":"8c60cb3bc9b20a50627d81741220f38a3e6ef1408ed999959c3fa38a14373648"} Nov 02 10:23:05.387113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.387094 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:05.388766 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.388740 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" event={"ID":"d099db5a-b4e0-47d9-8840-52d37187c172","Type":"ContainerStarted","Data":"08400333f1d8818118e189bd02f50552325800b2ada97b9f0270c026686d96d9"} Nov 02 10:23:05.388917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.388780 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" event={"ID":"d099db5a-b4e0-47d9-8840-52d37187c172","Type":"ContainerStarted","Data":"d518ce6797d1e4dbdfc5cda385adb92a190b9ed87c99d262b179485798abdb2e"} Nov 02 10:23:05.389056 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.389038 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:05.407448 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.407383 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" podStartSLOduration=1.407364123 podStartE2EDuration="1.407364123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:23:05.406148698 +0000 UTC m=+2864.246090695" watchObservedRunningTime="2026-11-02 10:23:05.407364123 +0000 UTC m=+2864.247306120" Nov 02 10:23:05.430353 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.430286 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" podStartSLOduration=1.430266489 podStartE2EDuration="1.430266489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:23:05.428513802 +0000 UTC m=+2864.268455797" watchObservedRunningTime="2026-11-02 10:23:05.430266489 +0000 UTC m=+2864.270208487" Nov 02 10:23:05.594961 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.594778 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt"] Nov 02 10:23:05.597174 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:23:05.597140 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71409ee6_b66e_4dcc_90f4_ebf2e0e53187.slice/crio-e4377dc20d72db1b696420c713617b1648fb212dcaf1687ee7299eafc122bdd7 WatchSource:0}: Error finding container e4377dc20d72db1b696420c713617b1648fb212dcaf1687ee7299eafc122bdd7: Status 404 returned error can't find the container with id e4377dc20d72db1b696420c713617b1648fb212dcaf1687ee7299eafc122bdd7 Nov 02 10:23:05.620150 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:05.620122 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:06.394758 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:06.394663 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" event={"ID":"71409ee6-b66e-4dcc-90f4-ebf2e0e53187","Type":"ContainerStarted","Data":"04bb3d4c9707f20c39d7f4127f3c92214eb0be44a27deefbd22e58773b8bc907"} Nov 02 10:23:06.394758 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:06.394719 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" event={"ID":"71409ee6-b66e-4dcc-90f4-ebf2e0e53187","Type":"ContainerStarted","Data":"e4377dc20d72db1b696420c713617b1648fb212dcaf1687ee7299eafc122bdd7"} Nov 02 10:23:06.395228 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:06.394914 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:06.415779 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:06.415730 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" podStartSLOduration=1.92519811 podStartE2EDuration="2.415715983s" totalImagesPullingTime="490.517873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:23:06.41440647 +0000 UTC m=+2865.254348469" watchObservedRunningTime="2026-11-02 10:23:06.415715983 +0000 UTC m=+2865.255658054" Nov 02 10:23:06.513526 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:06.513495 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:07.213862 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:07.213817 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:07.590486 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:07.590446 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:23:07.941792 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:07.941697 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:08.120061 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:08.120018 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:08.318391 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:08.318356 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:08.697781 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:08.697704 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:09.547323 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:09.547282 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:10.199728 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:10.199694 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:10.610179 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:10.610134 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:23:11.207518 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:11.207470 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:11.907460 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:11.907428 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:12.406074 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:12.406044 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:12.621088 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:12.621054 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:13.019907 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:13.019851 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:13.421543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:13.421510 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:14.108735 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:14.108703 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:14.357244 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:14.357213 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:14.536035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:14.536002 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:14.825079 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:14.825041 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:15.623747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:15.623711 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:16.397869 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:16.397835 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:16.427649 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:16.427612 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:17.219669 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:17.219639 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:17.402753 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:17.402718 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:17.997457 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:17.997375 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:18.719810 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:18.719778 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:19.412644 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:19.412606 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:20.109451 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:20.109414 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:20.815213 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:20.815182 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:21.522613 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:21.522581 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:22.226281 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:22.226255 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:23.021076 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:23.020995 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:23.318380 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:23.318348 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k"] Nov 02 10:23:23.338837 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:23.338676 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6"] Nov 02 10:23:23.483182 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:23.483147 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:23.738139 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:23.738052 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:23.915021 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:23.914990 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:24.267746 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.267715 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt"] Nov 02 10:23:24.269160 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.268000 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" podUID="71409ee6-b66e-4dcc-90f4-ebf2e0e53187" containerName="maas-api" containerID="cri-o://04bb3d4c9707f20c39d7f4127f3c92214eb0be44a27deefbd22e58773b8bc907" gracePeriod=30 Nov 02 10:23:24.425309 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.425272 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:24.502009 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.501973 4467 generic.go:356] "Generic (PLEG): container finished" podID="71409ee6-b66e-4dcc-90f4-ebf2e0e53187" containerID="04bb3d4c9707f20c39d7f4127f3c92214eb0be44a27deefbd22e58773b8bc907" exitCode=0 Nov 02 10:23:24.502225 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.502022 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" event={"ID":"71409ee6-b66e-4dcc-90f4-ebf2e0e53187","Type":"ContainerDied","Data":"04bb3d4c9707f20c39d7f4127f3c92214eb0be44a27deefbd22e58773b8bc907"} Nov 02 10:23:24.536507 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.536482 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:24.669447 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.669410 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:24.706617 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.706581 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-metrics-certs\") pod \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " Nov 02 10:23:24.706784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.706637 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-openshift-service-ca\") pod \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " Nov 02 10:23:24.706784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.706747 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls\") pod \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " Nov 02 10:23:24.706939 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.706784 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-kube-api-access-wstsv\" (UniqueName: \"kubernetes.io/projected/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-kube-api-access-wstsv\") pod \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\" (UID: \"71409ee6-b66e-4dcc-90f4-ebf2e0e53187\") " Nov 02 10:23:24.707129 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.707097 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-openshift-service-ca" pod "71409ee6-b66e-4dcc-90f4-ebf2e0e53187" (UID: "71409ee6-b66e-4dcc-90f4-ebf2e0e53187"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:23:24.709244 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.709216 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-metrics-certs" pod "71409ee6-b66e-4dcc-90f4-ebf2e0e53187" (UID: "71409ee6-b66e-4dcc-90f4-ebf2e0e53187"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:23:24.709244 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.709228 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-kube-api-access-wstsv" pod "71409ee6-b66e-4dcc-90f4-ebf2e0e53187" (UID: "71409ee6-b66e-4dcc-90f4-ebf2e0e53187"). InnerVolumeSpecName "kube-api-access-wstsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:23:24.709423 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.709406 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls" pod "71409ee6-b66e-4dcc-90f4-ebf2e0e53187" (UID: "71409ee6-b66e-4dcc-90f4-ebf2e0e53187"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:23:24.808592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.808557 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:24.808592 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.808594 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:24.808776 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.808605 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:24.808776 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.808613 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstsv\" (UniqueName: \"kubernetes.io/projected/71409ee6-b66e-4dcc-90f4-ebf2e0e53187-kube-api-access-wstsv\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:24.847931 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:24.847871 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:25.155626 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:25.155502 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:23:25.509743 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:25.509654 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" Nov 02 10:23:25.509743 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:25.509649 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt" event={"ID":"71409ee6-b66e-4dcc-90f4-ebf2e0e53187","Type":"ContainerDied","Data":"e4377dc20d72db1b696420c713617b1648fb212dcaf1687ee7299eafc122bdd7"} Nov 02 10:23:25.510303 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:25.509784 4467 scope.go:122] "RemoveContainer" containerID="04bb3d4c9707f20c39d7f4127f3c92214eb0be44a27deefbd22e58773b8bc907" Nov 02 10:23:25.536218 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:25.536186 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt"] Nov 02 10:23:25.537832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:25.537808 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8d45bb-7d69ccbd79-vd2xt"] Nov 02 10:23:26.721871 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:26.721834 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:23:26.855466 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:23:26.855430 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:23:27.331547 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:27.331513 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71409ee6-b66e-4dcc-90f4-ebf2e0e53187" path="/var/lib/kubelet/pods/71409ee6-b66e-4dcc-90f4-ebf2e0e53187/volumes" Nov 02 10:23:28.318956 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.318886 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" podUID="d099db5a-b4e0-47d9-8840-52d37187c172" containerName="payload-processing" containerID="cri-o://08400333f1d8818118e189bd02f50552325800b2ada97b9f0270c026686d96d9" gracePeriod=25 Nov 02 10:23:28.340341 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.340309 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" podUID="80631528-0928-4785-8cff-132d21aa9fbe" containerName="payload-pre-processing" containerID="cri-o://5d8de7b930705593208436079bfd1e456ef139275d7614d690352f92b7a9eb03" gracePeriod=25 Nov 02 10:23:28.532268 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.531354 4467 generic.go:356] "Generic (PLEG): container finished" podID="d099db5a-b4e0-47d9-8840-52d37187c172" containerID="08400333f1d8818118e189bd02f50552325800b2ada97b9f0270c026686d96d9" exitCode=0 Nov 02 10:23:28.532268 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.531473 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" event={"ID":"d099db5a-b4e0-47d9-8840-52d37187c172","Type":"ContainerDied","Data":"08400333f1d8818118e189bd02f50552325800b2ada97b9f0270c026686d96d9"} Nov 02 10:23:28.533272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.533012 4467 generic.go:356] "Generic (PLEG): container finished" podID="80631528-0928-4785-8cff-132d21aa9fbe" containerID="5d8de7b930705593208436079bfd1e456ef139275d7614d690352f92b7a9eb03" exitCode=0 Nov 02 10:23:28.533272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.533060 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" event={"ID":"80631528-0928-4785-8cff-132d21aa9fbe","Type":"ContainerDied","Data":"5d8de7b930705593208436079bfd1e456ef139275d7614d690352f92b7a9eb03"} Nov 02 10:23:28.605360 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.605330 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:28.638308 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.638284 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:28.746058 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.746021 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d099db5a-b4e0-47d9-8840-52d37187c172-kube-api-access-x8cgl\" (UniqueName: \"kubernetes.io/projected/d099db5a-b4e0-47d9-8840-52d37187c172-kube-api-access-x8cgl\") pod \"d099db5a-b4e0-47d9-8840-52d37187c172\" (UID: \"d099db5a-b4e0-47d9-8840-52d37187c172\") " Nov 02 10:23:28.746248 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.746092 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d099db5a-b4e0-47d9-8840-52d37187c172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d099db5a-b4e0-47d9-8840-52d37187c172-plugins-config-volume\") pod \"d099db5a-b4e0-47d9-8840-52d37187c172\" (UID: \"d099db5a-b4e0-47d9-8840-52d37187c172\") " Nov 02 10:23:28.746248 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.746134 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80631528-0928-4785-8cff-132d21aa9fbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80631528-0928-4785-8cff-132d21aa9fbe-plugins-config-volume\") pod \"80631528-0928-4785-8cff-132d21aa9fbe\" (UID: \"80631528-0928-4785-8cff-132d21aa9fbe\") " Nov 02 10:23:28.746248 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.746212 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80631528-0928-4785-8cff-132d21aa9fbe-kube-api-access-jrgq9\" (UniqueName: \"kubernetes.io/projected/80631528-0928-4785-8cff-132d21aa9fbe-kube-api-access-jrgq9\") pod \"80631528-0928-4785-8cff-132d21aa9fbe\" (UID: \"80631528-0928-4785-8cff-132d21aa9fbe\") " Nov 02 10:23:28.746558 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.746529 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80631528-0928-4785-8cff-132d21aa9fbe-plugins-config-volume" pod "80631528-0928-4785-8cff-132d21aa9fbe" (UID: "80631528-0928-4785-8cff-132d21aa9fbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:23:28.746645 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.746528 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d099db5a-b4e0-47d9-8840-52d37187c172-plugins-config-volume" pod "d099db5a-b4e0-47d9-8840-52d37187c172" (UID: "d099db5a-b4e0-47d9-8840-52d37187c172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:23:28.748112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.748091 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d099db5a-b4e0-47d9-8840-52d37187c172-kube-api-access-x8cgl" pod "d099db5a-b4e0-47d9-8840-52d37187c172" (UID: "d099db5a-b4e0-47d9-8840-52d37187c172"). InnerVolumeSpecName "kube-api-access-x8cgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:23:28.748315 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.748299 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80631528-0928-4785-8cff-132d21aa9fbe-kube-api-access-jrgq9" pod "80631528-0928-4785-8cff-132d21aa9fbe" (UID: "80631528-0928-4785-8cff-132d21aa9fbe"). InnerVolumeSpecName "kube-api-access-jrgq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:23:28.847833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.847759 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8cgl\" (UniqueName: \"kubernetes.io/projected/d099db5a-b4e0-47d9-8840-52d37187c172-kube-api-access-x8cgl\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:28.847833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.847786 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d099db5a-b4e0-47d9-8840-52d37187c172-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:28.847833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.847796 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80631528-0928-4785-8cff-132d21aa9fbe-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:28.847833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:28.847804 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrgq9\" (UniqueName: \"kubernetes.io/projected/80631528-0928-4785-8cff-132d21aa9fbe-kube-api-access-jrgq9\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:29.539751 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.539720 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" Nov 02 10:23:29.539751 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.539735 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k" event={"ID":"d099db5a-b4e0-47d9-8840-52d37187c172","Type":"ContainerDied","Data":"d518ce6797d1e4dbdfc5cda385adb92a190b9ed87c99d262b179485798abdb2e"} Nov 02 10:23:29.540333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.539807 4467 scope.go:122] "RemoveContainer" containerID="08400333f1d8818118e189bd02f50552325800b2ada97b9f0270c026686d96d9" Nov 02 10:23:29.541293 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.541265 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" event={"ID":"80631528-0928-4785-8cff-132d21aa9fbe","Type":"ContainerDied","Data":"8c60cb3bc9b20a50627d81741220f38a3e6ef1408ed999959c3fa38a14373648"} Nov 02 10:23:29.541387 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.541278 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6" Nov 02 10:23:29.548835 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.548815 4467 scope.go:122] "RemoveContainer" containerID="5d8de7b930705593208436079bfd1e456ef139275d7614d690352f92b7a9eb03" Nov 02 10:23:29.558585 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.558566 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6"] Nov 02 10:23:29.563595 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.563570 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8d45bb-6dd778t8d6"] Nov 02 10:23:29.575703 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.575672 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k"] Nov 02 10:23:29.577546 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:29.577525 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8d45bb-646b57d6bfgg2k"] Nov 02 10:23:31.332350 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:31.332307 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80631528-0928-4785-8cff-132d21aa9fbe" path="/var/lib/kubelet/pods/80631528-0928-4785-8cff-132d21aa9fbe/volumes" Nov 02 10:23:31.332772 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:31.332677 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d099db5a-b4e0-47d9-8840-52d37187c172" path="/var/lib/kubelet/pods/d099db5a-b4e0-47d9-8840-52d37187c172/volumes" Nov 02 10:23:38.062384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:38.062350 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:23:41.864332 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:41.864298 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh"] Nov 02 10:23:41.864735 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:41.864503 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" podUID="94af169c-dda1-4cce-acdd-eb7070b2c30e" containerName="manager" containerID="cri-o://28b5dca9d63b33f59621b61c96e722e113c0284008ae724c83b06a0df3bb4162" gracePeriod=10 Nov 02 10:23:42.688697 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.688667 4467 generic.go:356] "Generic (PLEG): container finished" podID="94af169c-dda1-4cce-acdd-eb7070b2c30e" containerID="28b5dca9d63b33f59621b61c96e722e113c0284008ae724c83b06a0df3bb4162" exitCode=0 Nov 02 10:23:42.688862 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.688738 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" event={"ID":"94af169c-dda1-4cce-acdd-eb7070b2c30e","Type":"ContainerDied","Data":"28b5dca9d63b33f59621b61c96e722e113c0284008ae724c83b06a0df3bb4162"} Nov 02 10:23:42.727529 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.727504 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 10:23:42.815022 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.814993 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94af169c-dda1-4cce-acdd-eb7070b2c30e-kube-api-access-txrww\" (UniqueName: \"kubernetes.io/projected/94af169c-dda1-4cce-acdd-eb7070b2c30e-kube-api-access-txrww\") pod \"94af169c-dda1-4cce-acdd-eb7070b2c30e\" (UID: \"94af169c-dda1-4cce-acdd-eb7070b2c30e\") " Nov 02 10:23:42.815204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.815122 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94af169c-dda1-4cce-acdd-eb7070b2c30e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94af169c-dda1-4cce-acdd-eb7070b2c30e-extensions-socket-volume\") pod \"94af169c-dda1-4cce-acdd-eb7070b2c30e\" (UID: \"94af169c-dda1-4cce-acdd-eb7070b2c30e\") " Nov 02 10:23:42.815480 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.815458 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94af169c-dda1-4cce-acdd-eb7070b2c30e-extensions-socket-volume" pod "94af169c-dda1-4cce-acdd-eb7070b2c30e" (UID: "94af169c-dda1-4cce-acdd-eb7070b2c30e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:23:42.817231 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.817206 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94af169c-dda1-4cce-acdd-eb7070b2c30e-kube-api-access-txrww" pod "94af169c-dda1-4cce-acdd-eb7070b2c30e" (UID: "94af169c-dda1-4cce-acdd-eb7070b2c30e"). InnerVolumeSpecName "kube-api-access-txrww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:23:42.916486 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.916460 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txrww\" (UniqueName: \"kubernetes.io/projected/94af169c-dda1-4cce-acdd-eb7070b2c30e-kube-api-access-txrww\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:42.916486 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:42.916486 4467 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94af169c-dda1-4cce-acdd-eb7070b2c30e-extensions-socket-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:23:43.694606 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:43.694525 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" Nov 02 10:23:43.694606 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:43.694551 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh" event={"ID":"94af169c-dda1-4cce-acdd-eb7070b2c30e","Type":"ContainerDied","Data":"966f7e43c8e25c91b775e3ee2df7d7c5e03e798e9773593b95400026ac17b994"} Nov 02 10:23:43.694606 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:43.694604 4467 scope.go:122] "RemoveContainer" containerID="28b5dca9d63b33f59621b61c96e722e113c0284008ae724c83b06a0df3bb4162" Nov 02 10:23:43.716072 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:43.716048 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh"] Nov 02 10:23:43.720913 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:43.720868 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5njvh"] Nov 02 10:23:45.330464 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:23:45.330426 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94af169c-dda1-4cce-acdd-eb7070b2c30e" path="/var/lib/kubelet/pods/94af169c-dda1-4cce-acdd-eb7070b2c30e/volumes" Nov 02 10:23:56.854170 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:23:56.854128 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:24:13.056513 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:13.056427 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:24:26.855152 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:24:26.855113 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:24:42.870450 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870417 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh"] Nov 02 10:24:42.870849 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870837 4467 cpu_manager.go:414] "containerMap: removing container" podUID="71409ee6-b66e-4dcc-90f4-ebf2e0e53187" containerName="maas-api" Nov 02 10:24:42.870906 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870849 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71409ee6-b66e-4dcc-90f4-ebf2e0e53187" containerName="maas-api" Nov 02 10:24:42.870906 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870882 4467 cpu_manager.go:414] "containerMap: removing container" podUID="d099db5a-b4e0-47d9-8840-52d37187c172" containerName="payload-processing" Nov 02 10:24:42.870906 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870901 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d099db5a-b4e0-47d9-8840-52d37187c172" containerName="payload-processing" Nov 02 10:24:42.871012 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870915 4467 cpu_manager.go:414] "containerMap: removing container" podUID="80631528-0928-4785-8cff-132d21aa9fbe" containerName="payload-pre-processing" Nov 02 10:24:42.871012 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870921 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80631528-0928-4785-8cff-132d21aa9fbe" containerName="payload-pre-processing" Nov 02 10:24:42.871012 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870933 4467 cpu_manager.go:414] "containerMap: removing container" podUID="94af169c-dda1-4cce-acdd-eb7070b2c30e" containerName="manager" Nov 02 10:24:42.871012 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.870938 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94af169c-dda1-4cce-acdd-eb7070b2c30e" containerName="manager" Nov 02 10:24:42.871148 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.871012 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="d099db5a-b4e0-47d9-8840-52d37187c172" containerName="payload-processing" Nov 02 10:24:42.871148 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.871023 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="80631528-0928-4785-8cff-132d21aa9fbe" containerName="payload-pre-processing" Nov 02 10:24:42.871148 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.871039 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="94af169c-dda1-4cce-acdd-eb7070b2c30e" containerName="manager" Nov 02 10:24:42.871148 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.871045 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="71409ee6-b66e-4dcc-90f4-ebf2e0e53187" containerName="maas-api" Nov 02 10:24:42.875879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.875859 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:42.879952 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.879928 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9mq74\"" Nov 02 10:24:42.887034 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.887005 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh"] Nov 02 10:24:42.994300 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.994262 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6th7\" (UniqueName: \"kubernetes.io/projected/75c92ed5-3499-49c1-931d-2991574d5985-kube-api-access-w6th7\") pod \"kuadrant-operator-controller-manager-678d87766f-mhvgh\" (UID: \"75c92ed5-3499-49c1-931d-2991574d5985\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:42.994485 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:42.994384 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75c92ed5-3499-49c1-931d-2991574d5985-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mhvgh\" (UID: \"75c92ed5-3499-49c1-931d-2991574d5985\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:43.095085 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.095042 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75c92ed5-3499-49c1-931d-2991574d5985-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mhvgh\" (UID: \"75c92ed5-3499-49c1-931d-2991574d5985\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:43.095282 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.095143 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6th7\" (UniqueName: \"kubernetes.io/projected/75c92ed5-3499-49c1-931d-2991574d5985-kube-api-access-w6th7\") pod \"kuadrant-operator-controller-manager-678d87766f-mhvgh\" (UID: \"75c92ed5-3499-49c1-931d-2991574d5985\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:43.095518 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.095497 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75c92ed5-3499-49c1-931d-2991574d5985-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mhvgh\" (UID: \"75c92ed5-3499-49c1-931d-2991574d5985\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:43.107828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.107794 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6th7\" (UniqueName: \"kubernetes.io/projected/75c92ed5-3499-49c1-931d-2991574d5985-kube-api-access-w6th7\") pod \"kuadrant-operator-controller-manager-678d87766f-mhvgh\" (UID: \"75c92ed5-3499-49c1-931d-2991574d5985\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:43.190386 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.190309 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:43.469854 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.469826 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh"] Nov 02 10:24:43.472259 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:24:43.472226 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c92ed5_3499_49c1_931d_2991574d5985.slice/crio-749e753783f4393945691571353d369bd1dcb6a5a9cd5c723e5a84c0df62ad86 WatchSource:0}: Error finding container 749e753783f4393945691571353d369bd1dcb6a5a9cd5c723e5a84c0df62ad86: Status 404 returned error can't find the container with id 749e753783f4393945691571353d369bd1dcb6a5a9cd5c723e5a84c0df62ad86 Nov 02 10:24:43.983415 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.983374 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" event={"ID":"75c92ed5-3499-49c1-931d-2991574d5985","Type":"ContainerStarted","Data":"e14b8f65bc857cdd80c63309b1d9178f06d8de634e63209c6418dbe756963018"} Nov 02 10:24:43.984064 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.983479 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" event={"ID":"75c92ed5-3499-49c1-931d-2991574d5985","Type":"ContainerStarted","Data":"749e753783f4393945691571353d369bd1dcb6a5a9cd5c723e5a84c0df62ad86"} Nov 02 10:24:43.984064 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:43.983512 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:44.010218 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:44.010166 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" podStartSLOduration=2.01015276 podStartE2EDuration="2.01015276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:24:44.008069578 +0000 UTC m=+2962.848011575" watchObservedRunningTime="2026-11-02 10:24:44.01015276 +0000 UTC m=+2962.850094755" Nov 02 10:24:54.990371 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:24:54.990340 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mhvgh" Nov 02 10:24:56.853973 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:24:56.853935 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:25:01.079340 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:01.079309 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:25:13.114224 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:13.114183 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:25:16.396743 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:16.396704 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:25:19.967295 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:19.967244 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:25:21.703066 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:21.703012 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:25:21.703757 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:21.703719 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:25:26.855104 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:25:26.855069 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:25:33.786044 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:33.785960 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:25:34.990297 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:34.990266 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s"] Nov 02 10:25:34.997447 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:34.997423 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.010013 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.009982 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s"] Nov 02 10:25:35.100543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.100507 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-metrics-certs\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.100543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.100548 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-openshift-service-ca\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.100759 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.100630 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-maas-api-tls\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.100759 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.100709 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27zs\" (UniqueName: \"kubernetes.io/projected/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-kube-api-access-g27zs\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.202176 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.202142 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-metrics-certs\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.202176 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.202179 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-openshift-service-ca\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.202378 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.202224 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-maas-api-tls\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.202378 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.202265 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27zs\" (UniqueName: \"kubernetes.io/projected/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-kube-api-access-g27zs\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.202974 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.202946 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-openshift-service-ca\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.204849 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.204823 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-metrics-certs\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.205074 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.205048 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-maas-api-tls\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.215348 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.215321 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27zs\" (UniqueName: \"kubernetes.io/projected/13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e-kube-api-access-g27zs\") pod \"maas-api-6647fcb89f-mrg9s\" (UID: \"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.315645 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.315610 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:35.592854 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:35.592762 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s"] Nov 02 10:25:35.597867 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:25:35.597828 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b772f7_e4dc_43a8_ac6f_bf3dcbb4962e.slice/crio-abcdf58c0f0310752a12627594a9139498a32e64959b58c4133e89d02ba735e8 WatchSource:0}: Error finding container abcdf58c0f0310752a12627594a9139498a32e64959b58c4133e89d02ba735e8: Status 404 returned error can't find the container with id abcdf58c0f0310752a12627594a9139498a32e64959b58c4133e89d02ba735e8 Nov 02 10:25:36.278063 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:36.278029 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" event={"ID":"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e","Type":"ContainerStarted","Data":"9401f460af89916d09ba0f6abde2cf4a11651c5fc05419451a98e8bb4bb32fda"} Nov 02 10:25:36.278476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:36.278084 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" event={"ID":"13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e","Type":"ContainerStarted","Data":"abcdf58c0f0310752a12627594a9139498a32e64959b58c4133e89d02ba735e8"} Nov 02 10:25:36.278476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:36.278191 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:36.299775 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:36.299716 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" podStartSLOduration=1.7564884520000001 podStartE2EDuration="2.299700798s" totalImagesPullingTime="543.212346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:25:36.297564437 +0000 UTC m=+3015.137506431" watchObservedRunningTime="2026-11-02 10:25:36.299700798 +0000 UTC m=+3015.139642794" Nov 02 10:25:41.796848 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:41.796813 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:25:42.287706 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:42.287678 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-mrg9s" Nov 02 10:25:42.340924 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:42.340870 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj"] Nov 02 10:25:42.341157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:42.341126 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" podUID="5e201e45-6561-430f-b58c-325fb4c615d8" containerName="maas-api" containerID="cri-o://9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94" gracePeriod=30 Nov 02 10:25:43.292807 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.292776 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:25:43.635321 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.635294 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 10:25:43.786175 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.786120 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e201e45-6561-430f-b58c-325fb4c615d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e201e45-6561-430f-b58c-325fb4c615d8-openshift-service-ca\") pod \"5e201e45-6561-430f-b58c-325fb4c615d8\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " Nov 02 10:25:43.786399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.786244 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-maas-api-tls\") pod \"5e201e45-6561-430f-b58c-325fb4c615d8\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " Nov 02 10:25:43.786399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.786342 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e201e45-6561-430f-b58c-325fb4c615d8-kube-api-access-6h77m\" (UniqueName: \"kubernetes.io/projected/5e201e45-6561-430f-b58c-325fb4c615d8-kube-api-access-6h77m\") pod \"5e201e45-6561-430f-b58c-325fb4c615d8\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " Nov 02 10:25:43.786399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.786375 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-metrics-certs\") pod \"5e201e45-6561-430f-b58c-325fb4c615d8\" (UID: \"5e201e45-6561-430f-b58c-325fb4c615d8\") " Nov 02 10:25:43.786588 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.786486 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e201e45-6561-430f-b58c-325fb4c615d8-openshift-service-ca" pod "5e201e45-6561-430f-b58c-325fb4c615d8" (UID: "5e201e45-6561-430f-b58c-325fb4c615d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:25:43.786726 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.786703 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e201e45-6561-430f-b58c-325fb4c615d8-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:25:43.788785 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.788753 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-metrics-certs" pod "5e201e45-6561-430f-b58c-325fb4c615d8" (UID: "5e201e45-6561-430f-b58c-325fb4c615d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:25:43.789071 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.788990 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-maas-api-tls" pod "5e201e45-6561-430f-b58c-325fb4c615d8" (UID: "5e201e45-6561-430f-b58c-325fb4c615d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:25:43.789162 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.789119 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e201e45-6561-430f-b58c-325fb4c615d8-kube-api-access-6h77m" pod "5e201e45-6561-430f-b58c-325fb4c615d8" (UID: "5e201e45-6561-430f-b58c-325fb4c615d8"). InnerVolumeSpecName "kube-api-access-6h77m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:25:43.888103 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.887997 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:25:43.888103 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.888039 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h77m\" (UniqueName: \"kubernetes.io/projected/5e201e45-6561-430f-b58c-325fb4c615d8-kube-api-access-6h77m\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:25:43.888103 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:43.888057 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e201e45-6561-430f-b58c-325fb4c615d8-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:25:44.319169 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.319138 4467 generic.go:356] "Generic (PLEG): container finished" podID="5e201e45-6561-430f-b58c-325fb4c615d8" containerID="9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94" exitCode=0 Nov 02 10:25:44.319694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.319200 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" event={"ID":"5e201e45-6561-430f-b58c-325fb4c615d8","Type":"ContainerDied","Data":"9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94"} Nov 02 10:25:44.319694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.319204 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" Nov 02 10:25:44.319694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.319246 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj" event={"ID":"5e201e45-6561-430f-b58c-325fb4c615d8","Type":"ContainerDied","Data":"3089771f5e6f055424e7231e505c909a78890e5738001ecad333787a492e829d"} Nov 02 10:25:44.319694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.319272 4467 scope.go:122] "RemoveContainer" containerID="9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94" Nov 02 10:25:44.329214 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.329187 4467 scope.go:122] "RemoveContainer" containerID="9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94" Nov 02 10:25:44.329566 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:25:44.329537 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94\": container with ID starting with 9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94 not found: ID does not exist" containerID="9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94" Nov 02 10:25:44.329670 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.329579 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94"} err="failed to get container status \"9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94\": rpc error: code = NotFound desc = could not find container \"9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94\": container with ID starting with 9b966369bec78e1d175663329e0d25b85414660a63e1142bd7a33bebb896cc94 not found: ID does not exist" Nov 02 10:25:44.348551 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.348502 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj"] Nov 02 10:25:44.353581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.353553 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:25:44.354350 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.354325 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-5rkrj"] Nov 02 10:25:44.869906 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:44.869872 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:25:45.372435 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:45.372396 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e201e45-6561-430f-b58c-325fb4c615d8" path="/var/lib/kubelet/pods/5e201e45-6561-430f-b58c-325fb4c615d8/volumes" Nov 02 10:25:47.774548 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:47.774507 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:25:53.719828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:53.719786 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:25:56.853752 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:25:56.853710 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:25:59.783566 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:25:59.783532 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:26:00.953370 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:26:00.953335 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:26:01.705200 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:26:01.705169 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:26:13.166440 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:26:13.166407 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:26:26.854141 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:26:26.854103 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:26:56.854665 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:26:56.854619 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:27:13.248316 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:27:13.248233 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:27:24.273725 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:27:24.273692 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:27:24.466080 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:27:24.466048 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:27:25.161825 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:27:25.161791 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:27:26.855323 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:27:26.855288 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:27:51.304216 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:27:51.304180 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:27:56.854958 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:27:56.854921 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:27:57.547259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:27:57.547226 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:28:13.273833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:28:13.273797 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:28:18.441299 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:28:18.441260 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:28:18.618367 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:28:18.618324 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:28:26.855235 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:28:26.855197 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:28:56.854447 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:28:56.854411 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:29:16.450111 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:29:16.450066 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:29:21.797713 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:29:21.797668 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:29:22.051856 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:29:22.051758 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:29:26.854540 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:29:26.854494 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:29:38.260801 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:29:38.260768 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:29:44.637543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:29:44.637507 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:29:56.853728 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:29:56.853694 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:30:13.430903 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:30:13.430849 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:30:26.853810 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:30:26.853774 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:30:56.853958 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:30:56.853923 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:31:08.438346 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:08.438311 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:31:10.349578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:10.349537 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:31:11.231420 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:11.231388 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd"] Nov 02 10:31:11.231639 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:11.231614 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" containerID="cri-o://ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d" gracePeriod=30 Nov 02 10:31:13.418399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:13.418362 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:31:16.487347 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.487319 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 10:31:16.634171 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634131 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-kube-api-access-bfn9h\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-kube-api-access-bfn9h\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634239 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-envoy\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634301 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-data\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634342 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-credential-socket\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634368 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d67c593-7285-4756-a5b3-43529b70ae5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d67c593-7285-4756-a5b3-43529b70ae5a-istiod-ca-cert\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634420 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-socket\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634446 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-token\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-token\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634536 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-podinfo\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634592 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-certs\") pod \"5d67c593-7285-4756-a5b3-43529b70ae5a\" (UID: \"5d67c593-7285-4756-a5b3-43529b70ae5a\") " Nov 02 10:31:16.634625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634607 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-data" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:16.634907 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634683 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-credential-socket" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:16.634907 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.634825 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-socket" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:16.635062 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.635043 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.635126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.635075 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.635126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.635090 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.635126 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.635044 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-certs" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:16.635300 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.635125 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d67c593-7285-4756-a5b3-43529b70ae5a-istiod-ca-cert" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:31:16.637780 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.636731 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-kube-api-access-bfn9h" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "kube-api-access-bfn9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:31:16.637780 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.637114 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-envoy" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:16.637780 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.637138 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-podinfo" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:31:16.637780 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.637542 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-token" pod "5d67c593-7285-4756-a5b3-43529b70ae5a" (UID: "5d67c593-7285-4756-a5b3-43529b70ae5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:31:16.736028 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.735991 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.736028 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.736021 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d67c593-7285-4756-a5b3-43529b70ae5a-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.736028 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.736032 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.736280 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.736045 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d67c593-7285-4756-a5b3-43529b70ae5a-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.736280 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.736054 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d67c593-7285-4756-a5b3-43529b70ae5a-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.736280 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.736064 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfn9h\" (UniqueName: \"kubernetes.io/projected/5d67c593-7285-4756-a5b3-43529b70ae5a-kube-api-access-bfn9h\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:16.954015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.953921 4467 generic.go:356] "Generic (PLEG): container finished" podID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerID="ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d" exitCode=0 Nov 02 10:31:16.954189 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.954020 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" Nov 02 10:31:16.954189 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.954046 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" event={"ID":"5d67c593-7285-4756-a5b3-43529b70ae5a","Type":"ContainerDied","Data":"ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d"} Nov 02 10:31:16.954189 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.954102 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd" event={"ID":"5d67c593-7285-4756-a5b3-43529b70ae5a","Type":"ContainerDied","Data":"f0b328ae11934e48044d264d70e1ef73359baf6e9de00e6ff42075b3909791d0"} Nov 02 10:31:16.954189 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.954131 4467 scope.go:122] "RemoveContainer" containerID="ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d" Nov 02 10:31:16.963758 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.963740 4467 scope.go:122] "RemoveContainer" containerID="ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d" Nov 02 10:31:16.964023 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:31:16.964005 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d\": container with ID starting with ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d not found: ID does not exist" containerID="ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d" Nov 02 10:31:16.964071 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.964031 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d"} err="failed to get container status \"ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d\": rpc error: code = NotFound desc = could not find container \"ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d\": container with ID starting with ec884826fb89e3128df8ad03667b834a4dee42026dd3e11feb65c2e0560df75d not found: ID does not exist" Nov 02 10:31:16.980785 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.980759 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd"] Nov 02 10:31:16.985496 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:16.985473 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7prvd"] Nov 02 10:31:17.337956 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:17.332669 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" path="/var/lib/kubelet/pods/5d67c593-7285-4756-a5b3-43529b70ae5a/volumes" Nov 02 10:31:19.357028 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:19.356997 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:31:20.150512 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:20.150452 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:31:26.073437 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:26.073391 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:31:26.250270 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:26.250233 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt"] Nov 02 10:31:26.250467 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:26.250443 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" containerID="cri-o://f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c" gracePeriod=30 Nov 02 10:31:26.251932 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:26.251901 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z"] Nov 02 10:31:26.252161 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:26.252139 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" containerID="cri-o://3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82" gracePeriod=30 Nov 02 10:31:26.854331 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:31:26.854294 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:31:31.507205 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.507172 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 10:31:31.542419 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.542391 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 10:31:31.573668 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573629 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-envoy\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.573879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573694 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-credential-socket\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.573879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573726 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-certs\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.573879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573759 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-token\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-token\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.573879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573840 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-kube-api-access-tw6wz\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-kube-api-access-tw6wz\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.573879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573870 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/507b6ee6-7885-4ac3-b559-3736307901b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b6ee6-7885-4ac3-b559-3736307901b2-istio-podinfo\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573916 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-kube-api-access-dg56n\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-kube-api-access-dg56n\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573962 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-istio-token\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-istio-token\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573993 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-certs\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.573999 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-credential-socket" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574035 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-socket\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574071 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507b6ee6-7885-4ac3-b559-3736307901b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b6ee6-7885-4ac3-b559-3736307901b2-istiod-ca-cert\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574100 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istiod-ca-cert\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574131 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-envoy\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.574195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574176 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-credential-socket\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.574733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574219 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-podinfo\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.574733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574313 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-socket\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.574733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574332 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-certs" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.574733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574342 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-data\") pod \"507b6ee6-7885-4ac3-b559-3736307901b2\" (UID: \"507b6ee6-7885-4ac3-b559-3736307901b2\") " Nov 02 10:31:31.574733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574416 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-data\") pod \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\" (UID: \"7d736f29-0f60-4d45-be3e-0ae7ef4efac5\") " Nov 02 10:31:31.574733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574548 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-data" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.574733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574648 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istiod-ca-cert" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:31:31.575144 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574831 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-data" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.575144 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.574984 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.575144 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.575021 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.575144 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.575035 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.575144 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.575049 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.575144 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.575062 4467 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-data\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.575483 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.575267 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-certs" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.575672 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.575583 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-socket" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.578952 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.575875 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507b6ee6-7885-4ac3-b559-3736307901b2-istiod-ca-cert" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.576998 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-envoy" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.577191 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-credential-socket" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.577337 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-kube-api-access-tw6wz" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "kube-api-access-tw6wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.577469 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-envoy" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.577514 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-socket" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.578181 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-istio-token" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.578176 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/507b6ee6-7885-4ac3-b559-3736307901b2-istio-podinfo" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.578608 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-token" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:31:31.579094 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.578911 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-podinfo" pod "7d736f29-0f60-4d45-be3e-0ae7ef4efac5" (UID: "7d736f29-0f60-4d45-be3e-0ae7ef4efac5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 10:31:31.579593 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.579405 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-kube-api-access-dg56n" pod "507b6ee6-7885-4ac3-b559-3736307901b2" (UID: "507b6ee6-7885-4ac3-b559-3736307901b2"). InnerVolumeSpecName "kube-api-access-dg56n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:31:31.675988 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.675946 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.675988 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.675984 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.675988 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.675994 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676004 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw6wz\" (UniqueName: \"kubernetes.io/projected/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-kube-api-access-tw6wz\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676015 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b6ee6-7885-4ac3-b559-3736307901b2-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676023 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg56n\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-kube-api-access-dg56n\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676033 4467 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507b6ee6-7885-4ac3-b559-3736307901b2-istio-token\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676042 4467 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676050 4467 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-workload-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676058 4467 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b6ee6-7885-4ac3-b559-3736307901b2-istiod-ca-cert\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676067 4467 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-envoy\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676075 4467 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b6ee6-7885-4ac3-b559-3736307901b2-credential-socket\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:31.676259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:31.676083 4467 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d736f29-0f60-4d45-be3e-0ae7ef4efac5-istio-podinfo\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:31:32.026576 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.026541 4467 generic.go:356] "Generic (PLEG): container finished" podID="507b6ee6-7885-4ac3-b559-3736307901b2" containerID="f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c" exitCode=0 Nov 02 10:31:32.026766 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.026583 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" Nov 02 10:31:32.026766 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.026626 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" event={"ID":"507b6ee6-7885-4ac3-b559-3736307901b2","Type":"ContainerDied","Data":"f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c"} Nov 02 10:31:32.026766 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.026680 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt" event={"ID":"507b6ee6-7885-4ac3-b559-3736307901b2","Type":"ContainerDied","Data":"165864375e0031b552d9e89477b6493a9d778fbf65276923c261c23a150573b1"} Nov 02 10:31:32.026766 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.026711 4467 scope.go:122] "RemoveContainer" containerID="f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c" Nov 02 10:31:32.028157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.028134 4467 generic.go:356] "Generic (PLEG): container finished" podID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerID="3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82" exitCode=0 Nov 02 10:31:32.028259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.028168 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" event={"ID":"7d736f29-0f60-4d45-be3e-0ae7ef4efac5","Type":"ContainerDied","Data":"3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82"} Nov 02 10:31:32.028259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.028183 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" Nov 02 10:31:32.028259 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.028218 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z" event={"ID":"7d736f29-0f60-4d45-be3e-0ae7ef4efac5","Type":"ContainerDied","Data":"b8ab137854cf5774be1f624faece2673b6921c00586dc623a89b7f213fdafa21"} Nov 02 10:31:32.036668 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.036590 4467 scope.go:122] "RemoveContainer" containerID="f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c" Nov 02 10:31:32.036995 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:31:32.036968 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c\": container with ID starting with f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c not found: ID does not exist" containerID="f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c" Nov 02 10:31:32.037067 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.037009 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c"} err="failed to get container status \"f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c\": rpc error: code = NotFound desc = could not find container \"f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c\": container with ID starting with f023bd677b002b19e07cc55933e899839c5d296f3ec0637971edb3e48e95029c not found: ID does not exist" Nov 02 10:31:32.037067 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.037029 4467 scope.go:122] "RemoveContainer" containerID="3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82" Nov 02 10:31:32.045451 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.045433 4467 scope.go:122] "RemoveContainer" containerID="3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82" Nov 02 10:31:32.045738 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:31:32.045712 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82\": container with ID starting with 3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82 not found: ID does not exist" containerID="3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82" Nov 02 10:31:32.045831 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.045744 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82"} err="failed to get container status \"3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82\": rpc error: code = NotFound desc = could not find container \"3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82\": container with ID starting with 3dd731109571c83e4553cefffca7f9f39c6b90e098d7f2e56732aafee2581d82 not found: ID does not exist" Nov 02 10:31:32.053498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.053475 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt"] Nov 02 10:31:32.058250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.058229 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qtkkt"] Nov 02 10:31:32.070287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.070260 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z"] Nov 02 10:31:32.074573 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:32.074551 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5qw4z"] Nov 02 10:31:33.331828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:33.331752 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" path="/var/lib/kubelet/pods/507b6ee6-7885-4ac3-b559-3736307901b2/volumes" Nov 02 10:31:33.332220 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:31:33.332200 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" path="/var/lib/kubelet/pods/7d736f29-0f60-4d45-be3e-0ae7ef4efac5/volumes" Nov 02 10:31:56.853952 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:31:56.853918 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:32:13.472675 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:32:13.472636 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:32:26.854249 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:32:26.854215 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:32:43.775181 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:32:43.775139 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:32:47.848371 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:32:47.848342 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:32:50.668212 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:32:50.668181 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:32:50.846272 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:32:50.846234 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:32:51.943924 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:32:51.943853 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:32:52.215010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:32:52.214870 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:32:52.371242 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:32:52.371206 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:32:56.854121 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:32:56.854081 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:33:02.106933 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:02.106868 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:33:02.299786 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:02.299755 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:33:04.033201 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:04.033116 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:33:10.191289 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:10.191235 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:33:11.396000 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.395962 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89"] Nov 02 10:33:11.396389 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396351 4467 cpu_manager.go:414] "containerMap: removing container" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" Nov 02 10:33:11.396389 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396361 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" Nov 02 10:33:11.396389 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396376 4467 cpu_manager.go:414] "containerMap: removing container" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" Nov 02 10:33:11.396389 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396381 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" Nov 02 10:33:11.396389 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396390 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5e201e45-6561-430f-b58c-325fb4c615d8" containerName="maas-api" Nov 02 10:33:11.396564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396395 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e201e45-6561-430f-b58c-325fb4c615d8" containerName="maas-api" Nov 02 10:33:11.396564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396405 4467 cpu_manager.go:414] "containerMap: removing container" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" Nov 02 10:33:11.396564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396409 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" Nov 02 10:33:11.396564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396474 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="507b6ee6-7885-4ac3-b559-3736307901b2" containerName="istio-proxy" Nov 02 10:33:11.396564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396483 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d736f29-0f60-4d45-be3e-0ae7ef4efac5" containerName="istio-proxy" Nov 02 10:33:11.396564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396489 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e201e45-6561-430f-b58c-325fb4c615d8" containerName="maas-api" Nov 02 10:33:11.396564 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.396495 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d67c593-7285-4756-a5b3-43529b70ae5a" containerName="istio-proxy" Nov 02 10:33:11.399662 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.399637 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.406351 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.406326 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 10:33:11.414382 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.414351 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89"] Nov 02 10:33:11.480483 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.480445 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.480687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.480570 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.480687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.480605 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.480687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.480645 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbp9n\" (UniqueName: \"kubernetes.io/projected/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-kube-api-access-nbp9n\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.480826 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.480744 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.480883 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.480829 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.480978 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.480877 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.480978 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.480954 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.481073 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.481049 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.489525 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.489500 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b2fjp\"" Nov 02 10:33:11.582517 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.582480 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.582696 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.582525 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.582696 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.582630 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.582696 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.582675 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbp9n\" (UniqueName: \"kubernetes.io/projected/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-kube-api-access-nbp9n\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.582696 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.582695 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.582957 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.582735 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.583039 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.582970 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.583039 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.582956 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.583039 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.583024 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.583205 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.583072 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.583205 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.583084 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.583205 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.583123 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.583384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.583281 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.583432 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.583416 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.585195 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.585169 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.585361 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.585345 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.595593 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.595563 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.595736 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.595717 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbp9n\" (UniqueName: \"kubernetes.io/projected/cc4a7396-fc1a-40d9-9500-19ecb03d23b8-kube-api-access-nbp9n\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-fgc89\" (UID: \"cc4a7396-fc1a-40d9-9500-19ecb03d23b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:11.713749 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:11.713660 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:12.000929 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:12.000880 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89"] Nov 02 10:33:12.003265 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:33:12.003239 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4a7396_fc1a_40d9_9500_19ecb03d23b8.slice/crio-7189b6f01e02056fbe57a3a7852b788e5ea778687c210cf7c78365035138b65c WatchSource:0}: Error finding container 7189b6f01e02056fbe57a3a7852b788e5ea778687c210cf7c78365035138b65c: Status 404 returned error can't find the container with id 7189b6f01e02056fbe57a3a7852b788e5ea778687c210cf7c78365035138b65c Nov 02 10:33:12.005377 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:12.005347 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:33:12.005497 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:12.005424 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:33:12.005497 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:12.005468 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:33:12.509907 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:12.509858 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" event={"ID":"cc4a7396-fc1a-40d9-9500-19ecb03d23b8","Type":"ContainerStarted","Data":"c615eaacf94df6babe03138f4bacc92b9f92ec1af26657590f95cc4bc17f880a"} Nov 02 10:33:12.510411 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:12.509963 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" event={"ID":"cc4a7396-fc1a-40d9-9500-19ecb03d23b8","Type":"ContainerStarted","Data":"7189b6f01e02056fbe57a3a7852b788e5ea778687c210cf7c78365035138b65c"} Nov 02 10:33:12.535267 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:12.535207 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podStartSLOduration=1.535187082 podStartE2EDuration="1.535187082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:33:12.5343821 +0000 UTC m=+3471.374324099" watchObservedRunningTime="2026-11-02 10:33:12.535187082 +0000 UTC m=+3471.375129078" Nov 02 10:33:12.713817 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:12.713778 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:13.539168 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:13.539137 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:33:13.716320 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:13.716285 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 10:33:13.716509 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:13.716336 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 10:33:14.714283 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:14.714243 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 10:33:14.714736 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:14.714305 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 10:33:15.715160 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:15.715118 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 10:33:15.715578 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:15.715200 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 10:33:16.714225 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:16.714184 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 10:33:16.714412 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:16.714258 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 10:33:17.714474 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:17.714433 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 10:33:17.714954 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:17.714507 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 10:33:18.715032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:18.714991 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 10:33:18.715537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:18.715054 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 10:33:19.714625 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:19.714589 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 10:33:19.714819 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:19.714641 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 10:33:21.433978 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:21.433948 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:21.434619 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:21.434397 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:21.435317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:21.435295 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" Nov 02 10:33:26.417974 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.417939 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw"] Nov 02 10:33:26.421566 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.421536 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.436148 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.436112 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw"] Nov 02 10:33:26.535044 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535009 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.535044 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535052 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.535319 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535158 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.535319 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535229 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5z8\" (UniqueName: \"kubernetes.io/projected/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-kube-api-access-rs5z8\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.535319 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535267 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.535498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535322 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.535498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535351 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.535498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535429 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.535498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.535456 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636139 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636102 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636315 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636156 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636315 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636269 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636443 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636335 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636443 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636353 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636443 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636394 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636443 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636413 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5z8\" (UniqueName: \"kubernetes.io/projected/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-kube-api-access-rs5z8\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636643 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636459 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636643 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636464 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636643 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636490 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636785 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636711 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.636942 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636913 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.637070 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.636954 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.637187 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.637170 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.638455 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.638435 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.638792 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.638772 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.648693 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.648668 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.649456 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.649439 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5z8\" (UniqueName: \"kubernetes.io/projected/16563487-59d4-4d8a-ad4d-e4d2c9fa557c-kube-api-access-rs5z8\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-l4czw\" (UID: \"16563487-59d4-4d8a-ad4d-e4d2c9fa557c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.738490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:26.738390 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:26.854769 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:33:26.854733 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:33:27.023406 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:27.023380 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw"] Nov 02 10:33:27.025448 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:33:27.025419 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16563487_59d4_4d8a_ad4d_e4d2c9fa557c.slice/crio-e5e7ff8dfb7ae0312b3219f5d96b4870afcb7e3fe6c87e772714da8490057274 WatchSource:0}: Error finding container e5e7ff8dfb7ae0312b3219f5d96b4870afcb7e3fe6c87e772714da8490057274: Status 404 returned error can't find the container with id e5e7ff8dfb7ae0312b3219f5d96b4870afcb7e3fe6c87e772714da8490057274 Nov 02 10:33:27.030765 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:27.030732 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:33:27.030825 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:27.030802 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:33:27.030860 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:27.030830 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:33:27.607832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:27.607795 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" event={"ID":"16563487-59d4-4d8a-ad4d-e4d2c9fa557c","Type":"ContainerStarted","Data":"fb800f1edb685a01675d70ff33efbd81c96968c3672fd4bed72f98fc0058f734"} Nov 02 10:33:27.608389 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:27.607850 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" event={"ID":"16563487-59d4-4d8a-ad4d-e4d2c9fa557c","Type":"ContainerStarted","Data":"e5e7ff8dfb7ae0312b3219f5d96b4870afcb7e3fe6c87e772714da8490057274"} Nov 02 10:33:27.634774 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:27.634720 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podStartSLOduration=1.634705226 podStartE2EDuration="1.634705226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:33:27.632200523 +0000 UTC m=+3486.472142518" watchObservedRunningTime="2026-11-02 10:33:27.634705226 +0000 UTC m=+3486.474647221" Nov 02 10:33:27.738694 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:27.738658 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:28.740555 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:28.740518 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 10:33:28.740971 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:28.740588 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 10:33:29.739191 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:29.739156 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 10:33:29.739384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:29.739210 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 10:33:30.739275 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:30.739237 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 10:33:30.739687 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:30.739293 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 10:33:31.739430 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:31.739392 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 10:33:31.739957 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:31.739453 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 10:33:32.739386 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:32.739348 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 10:33:32.739593 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:32.739408 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 10:33:33.739374 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:33.739334 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 10:33:33.739568 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:33.739398 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 10:33:34.739311 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:34.739271 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 10:33:34.739739 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:34.739341 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 10:33:36.391516 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:36.391485 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:36.392011 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:36.391767 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:36.392725 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:33:36.392707 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" Nov 02 10:33:56.855037 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:33:56.855000 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:34:13.596494 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:34:13.596458 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:34:26.854964 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:34:26.854924 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:34:44.981871 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:34:44.981817 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:34:45.491324 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:34:45.491287 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:34:51.075869 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:34:51.075838 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:34:56.853568 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:34:56.853531 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:35:11.559973 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.559934 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj"] Nov 02 10:35:11.564255 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.564221 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.578087 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.578057 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj"] Nov 02 10:35:11.663563 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.663512 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2174c08-7034-41d6-b383-20e50a1298ee-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.663784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.663642 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwk2b\" (UniqueName: \"kubernetes.io/projected/f2174c08-7034-41d6-b383-20e50a1298ee-kube-api-access-hwk2b\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.663784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.663669 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2174c08-7034-41d6-b383-20e50a1298ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.663938 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.663773 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.663938 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.663847 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.663938 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.663873 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.664113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.663949 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2174c08-7034-41d6-b383-20e50a1298ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.664113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.663968 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.664113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.664076 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765116 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765072 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2174c08-7034-41d6-b383-20e50a1298ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765116 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765116 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765255 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765399 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765319 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2174c08-7034-41d6-b383-20e50a1298ee-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765534 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765472 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwk2b\" (UniqueName: \"kubernetes.io/projected/f2174c08-7034-41d6-b383-20e50a1298ee-kube-api-access-hwk2b\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765534 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765520 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2174c08-7034-41d6-b383-20e50a1298ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765647 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765551 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765647 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765583 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765647 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765612 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765647 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765617 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.765875 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765691 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.766015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765992 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2174c08-7034-41d6-b383-20e50a1298ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.766075 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.765994 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.766118 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.766080 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.767939 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.767912 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2174c08-7034-41d6-b383-20e50a1298ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.768061 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.767994 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2174c08-7034-41d6-b383-20e50a1298ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.783358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.783323 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwk2b\" (UniqueName: \"kubernetes.io/projected/f2174c08-7034-41d6-b383-20e50a1298ee-kube-api-access-hwk2b\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.783358 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.783341 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2174c08-7034-41d6-b383-20e50a1298ee-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-d7szj\" (UID: \"f2174c08-7034-41d6-b383-20e50a1298ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:11.882643 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:11.882541 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:12.163511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:12.163469 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj"] Nov 02 10:35:12.166409 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:35:12.166376 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2174c08_7034_41d6_b383_20e50a1298ee.slice/crio-ad354b9a7a45fa65ddc83859135f62b69144711150830ea1041ccb12320be50b WatchSource:0}: Error finding container ad354b9a7a45fa65ddc83859135f62b69144711150830ea1041ccb12320be50b: Status 404 returned error can't find the container with id ad354b9a7a45fa65ddc83859135f62b69144711150830ea1041ccb12320be50b Nov 02 10:35:12.168557 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:12.168525 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:35:12.168679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:12.168593 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:35:12.168679 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:12.168625 4467 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 10:35:13.115567 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:13.115530 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" event={"ID":"f2174c08-7034-41d6-b383-20e50a1298ee","Type":"ContainerStarted","Data":"00243efb8a8b3d631720eca556ebc63c5108ebee798806e561ab923f7c0d81e0"} Nov 02 10:35:13.116026 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:13.115588 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" event={"ID":"f2174c08-7034-41d6-b383-20e50a1298ee","Type":"ContainerStarted","Data":"ad354b9a7a45fa65ddc83859135f62b69144711150830ea1041ccb12320be50b"} Nov 02 10:35:13.140164 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:13.140101 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podStartSLOduration=2.140087372 podStartE2EDuration="2.140087372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:35:13.139162142 +0000 UTC m=+3591.979104143" watchObservedRunningTime="2026-11-02 10:35:13.140087372 +0000 UTC m=+3591.980029367" Nov 02 10:35:13.707700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:13.707663 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:35:13.885838 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:13.885801 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:13.886240 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:13.886154 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 10:35:13.886451 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:13.886422 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 10:35:14.897915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:14.888343 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 10:35:14.897915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:14.888409 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 10:35:15.903923 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:15.900097 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 10:35:15.903923 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:15.900163 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 10:35:16.883391 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:16.883354 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 10:35:16.884077 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:16.884040 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 10:35:17.895926 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:17.894363 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 10:35:17.895926 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:17.894433 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 10:35:18.883320 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:18.883279 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 10:35:18.883500 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:18.883352 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 10:35:19.883082 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:19.883045 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 10:35:19.883495 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:19.883104 4467 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 10:35:21.363834 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:21.363794 4467 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:21.364383 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:21.364357 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:21.366125 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:35:21.366096 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" Nov 02 10:35:26.854656 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:35:26.854619 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:35:56.855011 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:35:56.854968 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:36:08.061537 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:08.061453 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:08.273139 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:08.273080 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:13.771339 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:13.771303 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:15.560102 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:15.560056 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:16.156331 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:16.156295 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:16.342398 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:16.342364 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:20.015776 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:20.015742 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:20.195318 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:20.195288 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:21.088696 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.088654 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm"] Nov 02 10:36:21.094104 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.094071 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.101443 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.101412 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ce5e7da4-serving-cert\"" Nov 02 10:36:21.110071 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.109994 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm"] Nov 02 10:36:21.172930 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.172878 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t"] Nov 02 10:36:21.176100 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.176075 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-openshift-service-ca\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.176274 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.176111 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-metrics-certs\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.176274 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.176222 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-maas-api-tls\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.176274 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.176250 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9x8j\" (UniqueName: \"kubernetes.io/projected/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-kube-api-access-x9x8j\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.176639 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.176625 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:21.189495 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.189462 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ce5e7da4\"" Nov 02 10:36:21.189954 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.189925 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ce5e7da4-dockercfg-q94fz\"" Nov 02 10:36:21.191427 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.191395 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t"] Nov 02 10:36:21.208320 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.208285 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:21.213074 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.213038 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm"] Nov 02 10:36:21.216978 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.216953 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:21.227378 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.227351 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm"] Nov 02 10:36:21.277278 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.277247 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm\" (UID: \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\") " pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:21.277467 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.277285 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-maas-api-tls\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.277467 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.277308 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9x8j\" (UniqueName: \"kubernetes.io/projected/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-kube-api-access-x9x8j\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.277467 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.277380 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpszz\" (UniqueName: \"kubernetes.io/projected/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-kube-api-access-vpszz\") pod \"payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm\" (UID: \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\") " pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:21.277467 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.277442 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6227f-5f2e-4d87-b579-e8e4e571f557-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t\" (UID: \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:21.277680 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.277516 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9z6\" (UniqueName: \"kubernetes.io/projected/ddf6227f-5f2e-4d87-b579-e8e4e571f557-kube-api-access-nd9z6\") pod \"payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t\" (UID: \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:21.277761 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.277742 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-openshift-service-ca\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.277798 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.277786 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-metrics-certs\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.278407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.278383 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-openshift-service-ca\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.280391 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.280368 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-metrics-certs\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.280497 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.280416 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-maas-api-tls\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.290180 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.290151 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9x8j\" (UniqueName: \"kubernetes.io/projected/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-kube-api-access-x9x8j\") pod \"maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.379056 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.378963 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm\" (UID: \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\") " pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:21.379056 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.379028 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpszz\" (UniqueName: \"kubernetes.io/projected/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-kube-api-access-vpszz\") pod \"payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm\" (UID: \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\") " pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:21.379292 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.379060 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6227f-5f2e-4d87-b579-e8e4e571f557-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t\" (UID: \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:21.379292 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.379104 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9z6\" (UniqueName: \"kubernetes.io/projected/ddf6227f-5f2e-4d87-b579-e8e4e571f557-kube-api-access-nd9z6\") pod \"payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t\" (UID: \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:21.381953 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.381933 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ce5e7da4\"" Nov 02 10:36:21.390115 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.390087 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm\" (UID: \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\") " pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:21.390292 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.390155 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6227f-5f2e-4d87-b579-e8e4e571f557-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t\" (UID: \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:21.392058 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.392035 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9z6\" (UniqueName: \"kubernetes.io/projected/ddf6227f-5f2e-4d87-b579-e8e4e571f557-kube-api-access-nd9z6\") pod \"payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t\" (UID: \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:21.392176 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.392098 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpszz\" (UniqueName: \"kubernetes.io/projected/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-kube-api-access-vpszz\") pod \"payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm\" (UID: \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\") " pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:21.417915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.417852 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:21.442098 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.442058 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:21.495221 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.495188 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ce5e7da4-dockercfg-q94fz\"" Nov 02 10:36:21.502270 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.502235 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:21.532387 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.532357 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:21.705208 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.704295 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm"] Nov 02 10:36:21.710027 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.709442 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 10:36:21.797783 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.797745 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t"] Nov 02 10:36:21.798226 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:36:21.798193 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf6227f_5f2e_4d87_b579_e8e4e571f557.slice/crio-bfb4c4fb8e667aa04b22e25fa589eb54de434e5182c8d6f9335ccbb99bcd5e90 WatchSource:0}: Error finding container bfb4c4fb8e667aa04b22e25fa589eb54de434e5182c8d6f9335ccbb99bcd5e90: Status 404 returned error can't find the container with id bfb4c4fb8e667aa04b22e25fa589eb54de434e5182c8d6f9335ccbb99bcd5e90 Nov 02 10:36:21.838225 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:21.838172 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm"] Nov 02 10:36:21.842099 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:36:21.842066 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a120a7c_08a6_4d9c_a6e9_cbcd0f87e3d9.slice/crio-b80b428537633d2084b8ba7159d3318c2b541f1e6562c2ecd0e8b6e2d97a92e9 WatchSource:0}: Error finding container b80b428537633d2084b8ba7159d3318c2b541f1e6562c2ecd0e8b6e2d97a92e9: Status 404 returned error can't find the container with id b80b428537633d2084b8ba7159d3318c2b541f1e6562c2ecd0e8b6e2d97a92e9 Nov 02 10:36:22.117870 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.117810 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:22.449811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.449713 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" event={"ID":"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9","Type":"ContainerStarted","Data":"b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319"} Nov 02 10:36:22.449811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.449784 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:22.449811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.449809 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" event={"ID":"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9","Type":"ContainerStarted","Data":"b80b428537633d2084b8ba7159d3318c2b541f1e6562c2ecd0e8b6e2d97a92e9"} Nov 02 10:36:22.451707 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.451673 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" event={"ID":"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6","Type":"ContainerStarted","Data":"5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29"} Nov 02 10:36:22.451857 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.451732 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" event={"ID":"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6","Type":"ContainerStarted","Data":"1e8da6ec4cdd83762eddb501113f1d7e807b6598460fc932669c5a0d1cdcd6b0"} Nov 02 10:36:22.451952 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.451934 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:22.453618 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.453594 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" event={"ID":"ddf6227f-5f2e-4d87-b579-e8e4e571f557","Type":"ContainerStarted","Data":"ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398"} Nov 02 10:36:22.453741 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.453638 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" event={"ID":"ddf6227f-5f2e-4d87-b579-e8e4e571f557","Type":"ContainerStarted","Data":"bfb4c4fb8e667aa04b22e25fa589eb54de434e5182c8d6f9335ccbb99bcd5e90"} Nov 02 10:36:22.453741 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.453724 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:22.468728 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.468670 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" podStartSLOduration=1.468654788 podStartE2EDuration="1.468654788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:36:22.467490591 +0000 UTC m=+3661.307432587" watchObservedRunningTime="2026-11-02 10:36:22.468654788 +0000 UTC m=+3661.308596782" Nov 02 10:36:22.488813 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.488756 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" podStartSLOduration=0.911896034 podStartE2EDuration="1.488740756s" totalImagesPullingTime="576.844722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:36:22.487262947 +0000 UTC m=+3661.327204943" watchObservedRunningTime="2026-11-02 10:36:22.488740756 +0000 UTC m=+3661.328682750" Nov 02 10:36:22.508464 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.508404 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" podStartSLOduration=1.508385587 podStartE2EDuration="1.508385587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:36:22.506275375 +0000 UTC m=+3661.346217362" watchObservedRunningTime="2026-11-02 10:36:22.508385587 +0000 UTC m=+3661.348327582" Nov 02 10:36:22.947350 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:22.947309 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:23.560320 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:23.560282 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:36:23.612964 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:23.612929 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:24.412188 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:24.412156 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:25.221772 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:25.221738 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:25.912559 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:25.912518 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:26.740884 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:26.740852 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:26.854929 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:36:26.854873 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:36:27.515158 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:27.515117 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:28.215150 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:28.215116 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:28.464997 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:28.464971 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:36:29.017160 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:29.017126 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:29.792964 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:29.792931 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:30.519380 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:30.519348 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:31.203293 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:31.203255 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:31.906820 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:31.906787 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:32.614621 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:32.614588 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:33.356596 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:33.356567 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:33.462791 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:33.462755 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:36:33.463021 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:33.462958 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:36:34.013865 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:34.013836 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:34.717229 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:34.717197 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:34.998746 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:34.998670 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:35.458698 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:35.458662 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:35.941587 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:35.941545 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:36.119460 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:36.119427 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:36.315270 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:36.315241 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:36.564491 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:36.564457 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:37.133699 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:37.133658 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:37.816868 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:37.816837 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:38.516585 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:38.516553 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:39.215749 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:39.215717 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:39.921724 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:39.921688 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:40.661060 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:40.661027 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:41.417714 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:41.417678 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:42.113953 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:42.113923 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:42.815793 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:42.815764 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:43.574827 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:43.574786 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:43.756105 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:43.756070 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:44.298197 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:44.298168 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:45.096377 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:45.096344 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:45.804362 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:45.804331 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:46.496989 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:46.496960 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:47.206141 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:47.206103 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:47.904831 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:47.904792 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:48.723041 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:48.723007 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:49.414183 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:49.414151 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:50.110307 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:50.110274 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:50.833516 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:50.833481 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:51.572905 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:51.572864 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:52.235987 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:52.235950 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:52.912107 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:52.912072 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:53.614073 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:53.614041 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:54.322357 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:54.322323 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:55.010937 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:55.010885 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:55.699221 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:55.699187 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:56.496962 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:56.496928 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:56.854828 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:36:56.854795 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:36:57.204161 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:57.204078 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:57.937740 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:57.937705 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:58.703686 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:58.703654 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:36:59.525701 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:36:59.525651 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:00.110979 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:00.110933 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:00.908300 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:00.908263 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:01.606454 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:01.606421 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:02.325691 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:02.325655 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:03.119759 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:03.119727 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:03.809222 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:03.809187 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:04.621088 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:04.621053 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:05.332433 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:05.332402 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:06.006759 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:06.006714 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:06.712681 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:06.712641 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:07.401125 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:07.401097 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:08.114407 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:08.114377 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:08.801496 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:08.801460 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:09.495606 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:09.495570 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:10.216006 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:10.215967 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:11.004119 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:11.004082 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:11.713210 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:11.713174 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:12.433114 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:12.433071 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:13.124307 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:13.124273 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:13.844014 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:13.843980 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:14.025406 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:14.025370 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:14.620185 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:14.620154 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:15.317957 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:15.317920 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:16.059326 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:16.059296 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:16.819543 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:16.819513 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:17.615483 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:17.615450 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:18.312581 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:18.312540 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:19.021599 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:19.021566 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:19.722867 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:19.722836 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:20.476868 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:20.476837 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:21.212513 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:21.212477 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:21.920384 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:21.920354 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:22.620403 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:22.620367 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:23.295356 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:23.295325 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:23.993915 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:23.993852 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:24.708868 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:24.708834 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:25.406594 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:25.406557 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:26.109484 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:26.109450 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:26.818032 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:26.818001 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:26.855141 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:37:26.855104 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:37:27.526071 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:27.526043 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:28.215257 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:28.215222 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:29.045630 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:29.045586 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:29.723726 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:29.723689 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:30.413210 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:30.413174 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:31.177458 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:31.177409 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:31.806293 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:31.806251 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:32.501317 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:32.501286 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:33.296002 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:33.295928 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:34.006045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:34.006014 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:34.712061 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:34.712027 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:35.404204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:35.404172 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:36.109164 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:36.109121 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:36.804155 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:36.804119 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:37.505661 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:37.505613 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:38.212458 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:38.212424 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:38.910180 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:38.910148 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:39.606297 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:39.606262 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:40.437002 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:40.436967 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:41.107997 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:41.107957 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:41.813775 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:41.813746 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:42.522258 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:42.522224 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:43.214375 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:43.214339 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:43.950695 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:43.950647 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:44.684239 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:44.684200 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:45.373827 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:45.373783 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:46.102326 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:46.102289 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:46.909708 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:46.909680 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:47.601648 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:47.601615 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:48.298561 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:48.298527 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:49.121644 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:49.121613 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:49.811410 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:49.811379 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:50.501448 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:50.501416 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:51.306483 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:51.306444 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:52.095021 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:52.094980 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:52.851742 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:52.851710 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:53.509806 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:53.509772 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:54.204573 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:54.204540 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:54.909012 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:54.908978 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:55.604138 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:55.604099 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:56.394010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:56.393981 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:56.854709 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:37:56.854676 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:37:57.111504 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:57.111417 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:57.827158 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:57.827120 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:58.508628 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:58.508597 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:59.208204 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:59.208167 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:37:59.875610 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:37:59.875572 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:00.056612 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:00.056571 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:00.619986 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:00.619951 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:01.421120 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:01.421084 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:02.151261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:02.151225 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:02.817104 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:02.817058 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:03.541879 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:03.541848 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:04.313121 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:04.313081 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:05.025114 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:05.025082 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:05.720722 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:05.720687 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:06.409281 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:06.409246 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:07.110086 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:07.110052 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:07.400820 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:07.400728 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:07.809508 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:07.809478 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:08.518730 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:08.518695 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:09.228549 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:09.228511 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:09.919747 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:09.919711 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:10.610777 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:10.610742 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:11.317811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:11.317775 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:12.014667 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:12.014635 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:12.717787 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:12.717754 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:13.480784 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:13.480754 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:13.549092 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:13.549057 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm"] Nov 02 10:38:13.571375 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:13.571344 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t"] Nov 02 10:38:13.729040 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:13.729012 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:13.928043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:13.928008 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:14.081530 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.081487 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm"] Nov 02 10:38:14.081759 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.081700 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" podUID="2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" containerName="maas-api" containerID="cri-o://5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29" gracePeriod=30 Nov 02 10:38:14.120410 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.120372 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:14.307616 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.299766 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:14.349466 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.349435 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:38:14.509085 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.508988 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-metrics-certs\") pod \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " Nov 02 10:38:14.509085 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.509042 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-kube-api-access-x9x8j\" (UniqueName: \"kubernetes.io/projected/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-kube-api-access-x9x8j\") pod \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " Nov 02 10:38:14.509085 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.509074 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-openshift-service-ca\") pod \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " Nov 02 10:38:14.509373 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.509095 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-maas-api-tls\") pod \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\" (UID: \"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6\") " Nov 02 10:38:14.509478 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.509441 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-openshift-service-ca" pod "2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" (UID: "2e84bb16-3f06-4d34-9012-2c7aa45aa5a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:38:14.511719 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.511688 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-maas-api-tls" pod "2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" (UID: "2e84bb16-3f06-4d34-9012-2c7aa45aa5a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:38:14.511829 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.511714 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-metrics-certs" pod "2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" (UID: "2e84bb16-3f06-4d34-9012-2c7aa45aa5a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 10:38:14.511829 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.511752 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-kube-api-access-x9x8j" pod "2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" (UID: "2e84bb16-3f06-4d34-9012-2c7aa45aa5a6"). InnerVolumeSpecName "kube-api-access-x9x8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:38:14.610580 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.610545 4467 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-metrics-certs\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:38:14.610580 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.610577 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9x8j\" (UniqueName: \"kubernetes.io/projected/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-kube-api-access-x9x8j\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:38:14.610580 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.610588 4467 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-openshift-service-ca\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:38:14.610832 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:14.610597 4467 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6-maas-api-tls\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:38:15.025833 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.025794 4467 generic.go:356] "Generic (PLEG): container finished" podID="2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" containerID="5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29" exitCode=0 Nov 02 10:38:15.026287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.025845 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" Nov 02 10:38:15.026287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.025872 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" event={"ID":"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6","Type":"ContainerDied","Data":"5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29"} Nov 02 10:38:15.026287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.025948 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm" event={"ID":"2e84bb16-3f06-4d34-9012-2c7aa45aa5a6","Type":"ContainerDied","Data":"1e8da6ec4cdd83762eddb501113f1d7e807b6598460fc932669c5a0d1cdcd6b0"} Nov 02 10:38:15.026287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.025972 4467 scope.go:122] "RemoveContainer" containerID="5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29" Nov 02 10:38:15.035760 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.035737 4467 scope.go:122] "RemoveContainer" containerID="5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29" Nov 02 10:38:15.036066 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:38:15.036047 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29\": container with ID starting with 5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29 not found: ID does not exist" containerID="5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29" Nov 02 10:38:15.036131 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.036076 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29"} err="failed to get container status \"5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29\": rpc error: code = NotFound desc = could not find container \"5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29\": container with ID starting with 5f5c7fd6ed7bb874327f82affd5b2b0465352417e079104db591d5d3a895ae29 not found: ID does not exist" Nov 02 10:38:15.051757 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.051155 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm"] Nov 02 10:38:15.058295 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.058270 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ce5e7da4-6c5586968-gdwrm"] Nov 02 10:38:15.331021 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.330985 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" path="/var/lib/kubelet/pods/2e84bb16-3f06-4d34-9012-2c7aa45aa5a6/volumes" Nov 02 10:38:15.858108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:15.858074 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:38:18.550221 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.550177 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" podUID="4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9" containerName="payload-processing" containerID="cri-o://b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319" gracePeriod=25 Nov 02 10:38:18.571672 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.571636 4467 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" podUID="ddf6227f-5f2e-4d87-b579-e8e4e571f557" containerName="payload-pre-processing" containerID="cri-o://ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398" gracePeriod=25 Nov 02 10:38:18.831446 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.831411 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:38:18.850476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.850450 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-kube-api-access-vpszz\" (UniqueName: \"kubernetes.io/projected/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-kube-api-access-vpszz\") pod \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\" (UID: \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\") " Nov 02 10:38:18.850628 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.850613 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-plugins-config-volume\") pod \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\" (UID: \"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9\") " Nov 02 10:38:18.851008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.850974 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-plugins-config-volume" pod "4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9" (UID: "4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:38:18.852732 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.852710 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-kube-api-access-vpszz" pod "4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9" (UID: "4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9"). InnerVolumeSpecName "kube-api-access-vpszz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:38:18.859505 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.859484 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:38:18.951908 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.951857 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf6227f-5f2e-4d87-b579-e8e4e571f557-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6227f-5f2e-4d87-b579-e8e4e571f557-plugins-config-volume\") pod \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\" (UID: \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\") " Nov 02 10:38:18.952080 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.952041 4467 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf6227f-5f2e-4d87-b579-e8e4e571f557-kube-api-access-nd9z6\" (UniqueName: \"kubernetes.io/projected/ddf6227f-5f2e-4d87-b579-e8e4e571f557-kube-api-access-nd9z6\") pod \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\" (UID: \"ddf6227f-5f2e-4d87-b579-e8e4e571f557\") " Nov 02 10:38:18.952246 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.952234 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:38:18.952285 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.952240 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf6227f-5f2e-4d87-b579-e8e4e571f557-plugins-config-volume" pod "ddf6227f-5f2e-4d87-b579-e8e4e571f557" (UID: "ddf6227f-5f2e-4d87-b579-e8e4e571f557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 10:38:18.952285 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.952249 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpszz\" (UniqueName: \"kubernetes.io/projected/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9-kube-api-access-vpszz\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:38:18.954104 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:18.954084 4467 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf6227f-5f2e-4d87-b579-e8e4e571f557-kube-api-access-nd9z6" pod "ddf6227f-5f2e-4d87-b579-e8e4e571f557" (UID: "ddf6227f-5f2e-4d87-b579-e8e4e571f557"). InnerVolumeSpecName "kube-api-access-nd9z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 10:38:19.049303 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.049270 4467 generic.go:356] "Generic (PLEG): container finished" podID="4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9" containerID="b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319" exitCode=0 Nov 02 10:38:19.049490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.049315 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" Nov 02 10:38:19.049490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.049356 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" event={"ID":"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9","Type":"ContainerDied","Data":"b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319"} Nov 02 10:38:19.049490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.049407 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm" event={"ID":"4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9","Type":"ContainerDied","Data":"b80b428537633d2084b8ba7159d3318c2b541f1e6562c2ecd0e8b6e2d97a92e9"} Nov 02 10:38:19.049490 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.049429 4467 scope.go:122] "RemoveContainer" containerID="b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319" Nov 02 10:38:19.050855 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.050833 4467 generic.go:356] "Generic (PLEG): container finished" podID="ddf6227f-5f2e-4d87-b579-e8e4e571f557" containerID="ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398" exitCode=0 Nov 02 10:38:19.050959 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.050922 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" event={"ID":"ddf6227f-5f2e-4d87-b579-e8e4e571f557","Type":"ContainerDied","Data":"ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398"} Nov 02 10:38:19.051027 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.050965 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" event={"ID":"ddf6227f-5f2e-4d87-b579-e8e4e571f557","Type":"ContainerDied","Data":"bfb4c4fb8e667aa04b22e25fa589eb54de434e5182c8d6f9335ccbb99bcd5e90"} Nov 02 10:38:19.051083 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.051036 4467 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t" Nov 02 10:38:19.056003 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.055981 4467 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd9z6\" (UniqueName: \"kubernetes.io/projected/ddf6227f-5f2e-4d87-b579-e8e4e571f557-kube-api-access-nd9z6\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:38:19.056105 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.056005 4467 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6227f-5f2e-4d87-b579-e8e4e571f557-plugins-config-volume\") on node \"ip-10-0-138-234.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 10:38:19.058733 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.058713 4467 scope.go:122] "RemoveContainer" containerID="b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319" Nov 02 10:38:19.059079 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:38:19.059012 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319\": container with ID starting with b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319 not found: ID does not exist" containerID="b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319" Nov 02 10:38:19.059079 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.059052 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319"} err="failed to get container status \"b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319\": rpc error: code = NotFound desc = could not find container \"b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319\": container with ID starting with b390370b6e06b1799d0f48a3930a29a2cb11a33f6ef240a8b2f63f8ccb2b8319 not found: ID does not exist" Nov 02 10:38:19.059079 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.059076 4467 scope.go:122] "RemoveContainer" containerID="ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398" Nov 02 10:38:19.070520 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.070505 4467 scope.go:122] "RemoveContainer" containerID="ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398" Nov 02 10:38:19.070802 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:38:19.070778 4467 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398\": container with ID starting with ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398 not found: ID does not exist" containerID="ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398" Nov 02 10:38:19.070862 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.070803 4467 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398"} err="failed to get container status \"ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398\": rpc error: code = NotFound desc = could not find container \"ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398\": container with ID starting with ebfb2f9571549ecfffc8464eb5fa7c4b4b61b0f1b32414dba4136573e7465398 not found: ID does not exist" Nov 02 10:38:19.080546 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.080523 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm"] Nov 02 10:38:19.084464 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.084444 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ce5e7da4-6687db7995-l2tvm"] Nov 02 10:38:19.095491 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.095469 4467 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t"] Nov 02 10:38:19.100149 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.100129 4467 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ce5e7da4-7749878675-xxb2t"] Nov 02 10:38:19.331444 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.331365 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9" path="/var/lib/kubelet/pods/4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9/volumes" Nov 02 10:38:19.331740 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:19.331728 4467 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf6227f-5f2e-4d87-b579-e8e4e571f557" path="/var/lib/kubelet/pods/ddf6227f-5f2e-4d87-b579-e8e4e571f557/volumes" Nov 02 10:38:20.064799 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:20.064767 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:20.248122 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:20.248093 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:20.591600 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:20.591555 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:20.965185 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:20.965108 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:23.887447 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:23.887410 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:38:26.860465 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:26.860394 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:26.933132 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:38:26.933058 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:38:27.050112 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:27.049485 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:38:54.387330 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:54.387293 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:38:56.856083 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:38:56.856044 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:38:57.796573 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:38:57.796527 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:39:13.870061 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:39:13.870024 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:39:15.096505 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:39:15.096464 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:39:20.249333 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:39:20.249284 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:39:26.854906 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:39:26.854857 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:39:42.665826 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:39:42.665790 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:39:49.788010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:39:49.787968 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:39:49.967394 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:39:49.967354 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:39:52.972591 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:39:52.972552 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:39:56.881632 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:39:56.881588 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:39:57.942157 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:39:57.942119 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:40:04.372549 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:04.372512 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:40:07.410035 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:07.409994 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:40:07.410705 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:07.410671 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:40:07.450913 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:07.449061 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:40:07.450913 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:07.449129 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:40:07.479006 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:07.478973 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:40:07.479224 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:07.479024 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:40:13.918279 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:13.918245 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:40:21.799943 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:21.799906 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:40:26.854621 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:40:26.854583 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:40:32.083864 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:32.083819 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:40:35.647970 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:35.647232 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:40:37.409048 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:37.409006 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-d7szj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:40:37.409671 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:37.409636 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-d7szj" podUID="f2174c08-7034-41d6-b383-20e50a1298ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:40:37.442068 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:37.442015 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:40:37.442476 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:37.442429 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:40:37.490494 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:37.490450 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:40:37.581365 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:37.581318 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:40:43.975211 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:43.975166 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:40:56.854486 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:40:56.854448 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:40:59.777200 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:40:59.777160 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:41:07.946498 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:07.946464 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:41:12.906481 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:12.906449 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:41:13.969708 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:13.969677 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:41:22.437043 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:22.437002 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-l4czw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:41:22.437731 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:22.437696 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-l4czw" podUID="16563487-59d4-4d8a-ad4d-e4d2c9fa557c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:41:22.489656 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:22.489048 4467 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-fgc89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 10:41:22.489656 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:22.489126 4467 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-fgc89" podUID="cc4a7396-fc1a-40d9-9500-19ecb03d23b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 10:41:26.854399 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:41:26.854358 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:41:33.259920 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:33.259564 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:41:34.654488 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:41:34.654460 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:41:56.854585 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:41:56.854547 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:42:02.933045 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:42:02.933000 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:42:14.104917 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:42:14.104094 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:42:26.853097 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:42:26.853056 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:42:30.491246 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:42:30.491203 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:42:30.674326 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:42:30.674296 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:42:51.789382 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:42:51.789337 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-52pxc"] Nov 02 10:42:56.854333 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:42:56.854295 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:43:11.791781 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:11.791750 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f755f6879-xh4h5_30722de2-3a23-49f9-9672-50446962e7e2/manager/0.log" Nov 02 10:43:12.684777 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:12.684750 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b555b5c85-sr2mh_c8787154-b48a-4aba-8c76-d1d9d0e434a2/manager/0.log" Nov 02 10:43:12.908250 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:12.908223 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2htfm_eadb4244-9782-409a-b4d2-e875d845c023/server/0.log" Nov 02 10:43:13.099300 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:13.099275 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d2kkp_77449e74-83c7-4592-bea4-48a7d20412d8/manager/0.log" Nov 02 10:43:13.318830 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:13.318800 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7shdp_925f1379-43e8-4594-ae67-ae2cf3bafab3/proxy/0.log" Nov 02 10:43:14.154639 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:14.154609 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:43:14.497462 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:14.497389 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6647fcb89f-mrg9s_13b772f7-e4dc-43a8-ac6f-bf3dcbb4962e/maas-api/0.log" Nov 02 10:43:18.097015 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:18.096980 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:43:18.274723 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:18.274689 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:43:20.836433 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:20.836399 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mhvgh_75c92ed5-3499-49c1-931d-2991574d5985/manager/0.log" Nov 02 10:43:21.089450 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:21.089373 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-52pxc_2506bfdc-b907-401c-ad87-d390372a5e08/limitador/0.log" Nov 02 10:43:22.750735 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:22.750704 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-x8mlt_cb0c1710-51cc-419a-84d3-f9e57666226c/kube-auth-proxy/0.log" Nov 02 10:43:23.665705 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:23.665678 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-d7szj_f2174c08-7034-41d6-b383-20e50a1298ee/istio-proxy/0.log" Nov 02 10:43:23.844106 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:23.844073 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fgc89_cc4a7396-fc1a-40d9-9500-19ecb03d23b8/istio-proxy/0.log" Nov 02 10:43:24.012645 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:24.012571 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-l4czw_16563487-59d4-4d8a-ad4d-e4d2c9fa557c/istio-proxy/0.log" Nov 02 10:43:24.912468 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:24.912438 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dbbcf55d8-n4d68_6fc09a83-a593-4e4e-b4c0-4921265e7ac1/payload-pre-processing/0.log" Nov 02 10:43:26.246538 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:26.246509 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-h557j_b4fd49f6-ec97-487c-aebe-369ad16468d7/main/0.log" Nov 02 10:43:26.606283 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:26.606252 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xg2gk_3162dd91-0c87-4098-9c76-16fe6ef4b17b/main/0.log" Nov 02 10:43:26.853393 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:43:26.853355 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:43:26.960913 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:26.960823 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8g4phf_91fbf792-c0c1-4431-8ea0-e288a4ec58c4/main/0.log" Nov 02 10:43:38.068369 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068322 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jtjww/must-gather-9vcn5"] Nov 02 10:43:38.068786 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068772 4467 cpu_manager.go:414] "containerMap: removing container" podUID="2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" containerName="maas-api" Nov 02 10:43:38.068838 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068787 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" containerName="maas-api" Nov 02 10:43:38.068838 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068816 4467 cpu_manager.go:414] "containerMap: removing container" podUID="ddf6227f-5f2e-4d87-b579-e8e4e571f557" containerName="payload-pre-processing" Nov 02 10:43:38.068838 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068821 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf6227f-5f2e-4d87-b579-e8e4e571f557" containerName="payload-pre-processing" Nov 02 10:43:38.068838 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068832 4467 cpu_manager.go:414] "containerMap: removing container" podUID="4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9" containerName="payload-processing" Nov 02 10:43:38.068995 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068840 4467 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9" containerName="payload-processing" Nov 02 10:43:38.068995 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068937 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf6227f-5f2e-4d87-b579-e8e4e571f557" containerName="payload-pre-processing" Nov 02 10:43:38.068995 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068945 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a120a7c-08a6-4d9c-a6e9-cbcd0f87e3d9" containerName="payload-processing" Nov 02 10:43:38.068995 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.068955 4467 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e84bb16-3f06-4d34-9012-2c7aa45aa5a6" containerName="maas-api" Nov 02 10:43:38.072264 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.072244 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtjww/must-gather-9vcn5" Nov 02 10:43:38.079464 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.078375 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jtjww\"/\"openshift-service-ca.crt\"" Nov 02 10:43:38.079647 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.079624 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jtjww\"/\"default-dockercfg-frcvk\"" Nov 02 10:43:38.079762 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.079744 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jtjww\"/\"kube-root-ca.crt\"" Nov 02 10:43:38.082710 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.082687 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtjww/must-gather-9vcn5"] Nov 02 10:43:38.110811 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.110785 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cf99cfa-5722-421f-8f0a-4b19417a4786-must-gather-output\") pod \"must-gather-9vcn5\" (UID: \"3cf99cfa-5722-421f-8f0a-4b19417a4786\") " pod="openshift-must-gather-jtjww/must-gather-9vcn5" Nov 02 10:43:38.110930 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.110827 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxmsl\" (UniqueName: \"kubernetes.io/projected/3cf99cfa-5722-421f-8f0a-4b19417a4786-kube-api-access-wxmsl\") pod \"must-gather-9vcn5\" (UID: \"3cf99cfa-5722-421f-8f0a-4b19417a4786\") " pod="openshift-must-gather-jtjww/must-gather-9vcn5" Nov 02 10:43:38.194559 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.194530 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:43:38.211880 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.211858 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cf99cfa-5722-421f-8f0a-4b19417a4786-must-gather-output\") pod \"must-gather-9vcn5\" (UID: \"3cf99cfa-5722-421f-8f0a-4b19417a4786\") " pod="openshift-must-gather-jtjww/must-gather-9vcn5" Nov 02 10:43:38.212008 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.211917 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxmsl\" (UniqueName: \"kubernetes.io/projected/3cf99cfa-5722-421f-8f0a-4b19417a4786-kube-api-access-wxmsl\") pod \"must-gather-9vcn5\" (UID: \"3cf99cfa-5722-421f-8f0a-4b19417a4786\") " pod="openshift-must-gather-jtjww/must-gather-9vcn5" Nov 02 10:43:38.212205 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.212186 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cf99cfa-5722-421f-8f0a-4b19417a4786-must-gather-output\") pod \"must-gather-9vcn5\" (UID: \"3cf99cfa-5722-421f-8f0a-4b19417a4786\") " pod="openshift-must-gather-jtjww/must-gather-9vcn5" Nov 02 10:43:38.224261 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.224236 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxmsl\" (UniqueName: \"kubernetes.io/projected/3cf99cfa-5722-421f-8f0a-4b19417a4786-kube-api-access-wxmsl\") pod \"must-gather-9vcn5\" (UID: \"3cf99cfa-5722-421f-8f0a-4b19417a4786\") " pod="openshift-must-gather-jtjww/must-gather-9vcn5" Nov 02 10:43:38.388209 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.388137 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtjww/must-gather-9vcn5" Nov 02 10:43:38.452573 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.452544 4467 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" limit=1024 Nov 02 10:43:38.663277 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.663254 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtjww/must-gather-9vcn5"] Nov 02 10:43:38.665658 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:43:38.665626 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cf99cfa_5722_421f_8f0a_4b19417a4786.slice/crio-b7781afe2ded4a87794f7b342f17f4ea96fe3dd6a32a91f0168966eb305f5464 WatchSource:0}: Error finding container b7781afe2ded4a87794f7b342f17f4ea96fe3dd6a32a91f0168966eb305f5464: Status 404 returned error can't find the container with id b7781afe2ded4a87794f7b342f17f4ea96fe3dd6a32a91f0168966eb305f5464 Nov 02 10:43:38.667795 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:38.667778 4467 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 10:43:39.077718 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:39.077684 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtjww/must-gather-9vcn5" event={"ID":"3cf99cfa-5722-421f-8f0a-4b19417a4786","Type":"ContainerStarted","Data":"b7781afe2ded4a87794f7b342f17f4ea96fe3dd6a32a91f0168966eb305f5464"} Nov 02 10:43:41.772088 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:41.772049 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zxrg9_568d527d-7b27-4e41-a946-1893e04d2653/global-pull-secret-syncer/0.log" Nov 02 10:43:41.945927 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:41.945877 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8t5nd_dc719dc6-3e71-4f85-b7fc-51cdee604e74/konnectivity-agent/0.log" Nov 02 10:43:42.130828 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:42.130792 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-234.us-east-2.compute.internal_7ba5c7edc9657ad596c668f03e09721a/haproxy/0.log" Nov 02 10:43:45.113080 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:45.113042 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtjww/must-gather-9vcn5" event={"ID":"3cf99cfa-5722-421f-8f0a-4b19417a4786","Type":"ContainerStarted","Data":"a29e62702388d20a4222b47cef527586159b0bf93101c7fb47c63272c4a0ab50"} Nov 02 10:43:45.113582 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:45.113098 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtjww/must-gather-9vcn5" event={"ID":"3cf99cfa-5722-421f-8f0a-4b19417a4786","Type":"ContainerStarted","Data":"2c7e62bd2df794975cd2d1ba9c7c0178bf81a408cafec90398bd294e584673da"} Nov 02 10:43:45.133950 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:45.133879 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jtjww/must-gather-9vcn5" podStartSLOduration=1.5284445039999999 podStartE2EDuration="7.133865431s" totalImagesPullingTime="5.605420927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:43:45.131733748 +0000 UTC m=+4103.971675744" watchObservedRunningTime="2026-11-02 10:43:45.133865431 +0000 UTC m=+4103.973807456" Nov 02 10:43:45.416085 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:45.416010 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f755f6879-xh4h5_30722de2-3a23-49f9-9672-50446962e7e2/manager/0.log" Nov 02 10:43:45.994700 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:45.994661 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b555b5c85-sr2mh_c8787154-b48a-4aba-8c76-d1d9d0e434a2/manager/0.log" Nov 02 10:43:46.011505 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:46.011475 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2htfm_eadb4244-9782-409a-b4d2-e875d845c023/server/0.log" Nov 02 10:43:46.689010 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:46.688977 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d2kkp_77449e74-83c7-4592-bea4-48a7d20412d8/manager/0.log" Nov 02 10:43:46.706095 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:46.706070 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7shdp_925f1379-43e8-4594-ae67-ae2cf3bafab3/proxy/0.log" Nov 02 10:43:47.530618 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:47.530583 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f755f6879-xh4h5_30722de2-3a23-49f9-9672-50446962e7e2/manager/0.log" Nov 02 10:43:48.134481 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:48.134442 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b555b5c85-sr2mh_c8787154-b48a-4aba-8c76-d1d9d0e434a2/manager/0.log" Nov 02 10:43:48.153018 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:48.152995 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2htfm_eadb4244-9782-409a-b4d2-e875d845c023/server/0.log" Nov 02 10:43:48.945855 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:48.945811 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d2kkp_77449e74-83c7-4592-bea4-48a7d20412d8/manager/0.log" Nov 02 10:43:48.985591 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:48.985552 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7shdp_925f1379-43e8-4594-ae67-ae2cf3bafab3/proxy/0.log" Nov 02 10:43:49.592987 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:49.592955 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mhvgh_75c92ed5-3499-49c1-931d-2991574d5985/manager/0.log" Nov 02 10:43:49.627279 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:49.627247 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-52pxc_2506bfdc-b907-401c-ad87-d390372a5e08/limitador/0.log" Nov 02 10:43:49.899511 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:49.899437 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f755f6879-xh4h5_30722de2-3a23-49f9-9672-50446962e7e2/manager/0.log" Nov 02 10:43:50.460296 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:50.460239 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b555b5c85-sr2mh_c8787154-b48a-4aba-8c76-d1d9d0e434a2/manager/0.log" Nov 02 10:43:50.474826 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:50.474800 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2htfm_eadb4244-9782-409a-b4d2-e875d845c023/server/0.log" Nov 02 10:43:51.129440 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.129410 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d2kkp_77449e74-83c7-4592-bea4-48a7d20412d8/manager/0.log" Nov 02 10:43:51.177802 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.177774 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7shdp_925f1379-43e8-4594-ae67-ae2cf3bafab3/proxy/0.log" Nov 02 10:43:51.642119 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.642093 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rg9l4_f6ec3b03-947f-4658-a099-f3957b455739/kube-state-metrics/0.log" Nov 02 10:43:51.667615 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.667587 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rg9l4_f6ec3b03-947f-4658-a099-f3957b455739/kube-rbac-proxy-main/0.log" Nov 02 10:43:51.697231 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.697200 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rg9l4_f6ec3b03-947f-4658-a099-f3957b455739/kube-rbac-proxy-self/0.log" Nov 02 10:43:51.729559 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.729526 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5994556fdb-hmv5q_ca0d1336-1b3f-45dd-b201-1bf53156fe92/metrics-server/0.log" Nov 02 10:43:51.767062 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.767021 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-2hvrl_c741a9ba-17b1-48fc-9d63-44938bb816eb/monitoring-plugin/0.log" Nov 02 10:43:51.799935 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.799664 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7ct4q_b25b935c-ea6d-472f-a465-bb9b626c5365/node-exporter/0.log" Nov 02 10:43:51.833344 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.833317 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7ct4q_b25b935c-ea6d-472f-a465-bb9b626c5365/kube-rbac-proxy/0.log" Nov 02 10:43:51.861555 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:51.861531 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7ct4q_b25b935c-ea6d-472f-a465-bb9b626c5365/init-textfile/0.log" Nov 02 10:43:52.166001 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.165969 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ed1e880-0148-4f30-827f-5da46b91c2cf/prometheus/0.log" Nov 02 10:43:52.197203 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.197173 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ed1e880-0148-4f30-827f-5da46b91c2cf/config-reloader/0.log" Nov 02 10:43:52.233707 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.233685 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ed1e880-0148-4f30-827f-5da46b91c2cf/thanos-sidecar/0.log" Nov 02 10:43:52.240470 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.240441 4467 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s"] Nov 02 10:43:52.249529 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.249503 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.257002 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.256971 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s"] Nov 02 10:43:52.257269 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.257249 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwdzc\"/\"openshift-service-ca.crt\"" Nov 02 10:43:52.257366 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.257348 4467 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwdzc\"/\"kube-root-ca.crt\"" Nov 02 10:43:52.257459 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.257443 4467 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zwdzc\"/\"default-dockercfg-ktm6x\"" Nov 02 10:43:52.270601 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.270580 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ed1e880-0148-4f30-827f-5da46b91c2cf/kube-rbac-proxy-web/0.log" Nov 02 10:43:52.298162 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.298104 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ed1e880-0148-4f30-827f-5da46b91c2cf/kube-rbac-proxy/0.log" Nov 02 10:43:52.331336 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.331312 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ed1e880-0148-4f30-827f-5da46b91c2cf/kube-rbac-proxy-thanos/0.log" Nov 02 10:43:52.350072 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.350045 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-lib-modules\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.350232 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.350090 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-sys\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.350232 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.350218 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6st\" (UniqueName: \"kubernetes.io/projected/5028a470-9169-4094-990f-c82ce261171f-kube-api-access-fk6st\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.350360 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.350260 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-podres\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.350360 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.350281 4467 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-proc\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.357923 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.357873 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ed1e880-0148-4f30-827f-5da46b91c2cf/init-config-reloader/0.log" Nov 02 10:43:52.392357 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.392318 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-x2bp8_c364e8ee-e971-43c9-ab10-1e92f7ccddf8/prometheus-operator/0.log" Nov 02 10:43:52.417554 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.417460 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-x2bp8_c364e8ee-e971-43c9-ab10-1e92f7ccddf8/kube-rbac-proxy/0.log" Nov 02 10:43:52.450194 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.450143 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-q2q2b_903de960-6f67-4baa-95ac-6a6712998fc4/prometheus-operator-admission-webhook/0.log" Nov 02 10:43:52.450970 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.450881 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-sys\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.451113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.451002 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-sys\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.451113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.451016 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6st\" (UniqueName: \"kubernetes.io/projected/5028a470-9169-4094-990f-c82ce261171f-kube-api-access-fk6st\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.451113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.451066 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-podres\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.451113 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.451095 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-proc\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.451414 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.451139 4467 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-lib-modules\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.451414 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.451174 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-podres\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.451414 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.451249 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-proc\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.451414 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.451280 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5028a470-9169-4094-990f-c82ce261171f-lib-modules\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.463672 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.463642 4467 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6st\" (UniqueName: \"kubernetes.io/projected/5028a470-9169-4094-990f-c82ce261171f-kube-api-access-fk6st\") pod \"perf-node-gather-daemonset-f7b2s\" (UID: \"5028a470-9169-4094-990f-c82ce261171f\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.569353 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.569319 4467 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:52.847425 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:52.847392 4467 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s"] Nov 02 10:43:52.851271 ip-10-0-138-234 kubenswrapper[4467]: W1102 10:43:52.851246 4467 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5028a470_9169_4094_990f_c82ce261171f.slice/crio-ed950bdeb59f2ae335060ac60f59e235af5a699ef958698206240f753f6470d1 WatchSource:0}: Error finding container ed950bdeb59f2ae335060ac60f59e235af5a699ef958698206240f753f6470d1: Status 404 returned error can't find the container with id ed950bdeb59f2ae335060ac60f59e235af5a699ef958698206240f753f6470d1 Nov 02 10:43:53.125001 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.124927 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-x8mlt_cb0c1710-51cc-419a-84d3-f9e57666226c/kube-auth-proxy/0.log" Nov 02 10:43:53.159661 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.159624 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" event={"ID":"5028a470-9169-4094-990f-c82ce261171f","Type":"ContainerStarted","Data":"5377b0dbfa5ad4fc45ae06615babe0626709386e9d45544b2c1a1f4541b89c68"} Nov 02 10:43:53.159827 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.159682 4467 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" event={"ID":"5028a470-9169-4094-990f-c82ce261171f","Type":"ContainerStarted","Data":"ed950bdeb59f2ae335060ac60f59e235af5a699ef958698206240f753f6470d1"} Nov 02 10:43:53.159880 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.159840 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:43:53.178255 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.178211 4467 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" podStartSLOduration=1.1781971900000001 podStartE2EDuration="1.17819719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 10:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:43:53.177040038 +0000 UTC m=+4112.016982034" watchObservedRunningTime="2026-11-02 10:43:53.17819719 +0000 UTC m=+4112.018139211" Nov 02 10:43:53.225354 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.225326 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-d7szj_f2174c08-7034-41d6-b383-20e50a1298ee/istio-proxy/0.log" Nov 02 10:43:53.243014 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.242984 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fgc89_cc4a7396-fc1a-40d9-9500-19ecb03d23b8/istio-proxy/0.log" Nov 02 10:43:53.260938 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.260909 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-l4czw_16563487-59d4-4d8a-ad4d-e4d2c9fa557c/istio-proxy/0.log" Nov 02 10:43:53.358299 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:53.358272 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dbbcf55d8-n4d68_6fc09a83-a593-4e4e-b4c0-4921265e7ac1/payload-pre-processing/0.log" Nov 02 10:43:56.643108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:56.643071 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mhvgh_75c92ed5-3499-49c1-931d-2991574d5985/manager/0.log" Nov 02 10:43:56.665122 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:56.665096 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-52pxc_2506bfdc-b907-401c-ad87-d390372a5e08/limitador/0.log" Nov 02 10:43:56.862307 ip-10-0-138-234 kubenswrapper[4467]: E1102 10:43:56.862255 4467 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cs7kk" containerName="extractor" Nov 02 10:43:57.056495 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:57.056420 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fffwx_8790c677-2d64-4973-a7e5-9ba0c04cc22d/dns/0.log" Nov 02 10:43:57.092584 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:57.092552 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fffwx_8790c677-2d64-4973-a7e5-9ba0c04cc22d/kube-rbac-proxy/0.log" Nov 02 10:43:57.241944 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:57.241913 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4xcxp_aad130bd-a2d7-40a9-8d1e-d6a728b7d49e/dns-node-resolver/0.log" Nov 02 10:43:57.909782 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:57.909124 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6kvjb_06c7eba7-d115-4796-89a5-47a4b7af0789/node-ca/0.log" Nov 02 10:43:59.175239 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:43:59.175211 4467 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-f7b2s" Nov 02 10:44:00.160307 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:00.160273 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-x8mlt_cb0c1710-51cc-419a-84d3-f9e57666226c/kube-auth-proxy/0.log" Nov 02 10:44:00.327257 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:00.327228 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-d7szj_f2174c08-7034-41d6-b383-20e50a1298ee/istio-proxy/0.log" Nov 02 10:44:00.360149 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:00.360113 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fgc89_cc4a7396-fc1a-40d9-9500-19ecb03d23b8/istio-proxy/0.log" Nov 02 10:44:00.397405 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:00.396997 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-l4czw_16563487-59d4-4d8a-ad4d-e4d2c9fa557c/istio-proxy/0.log" Nov 02 10:44:00.561219 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:00.561157 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dbbcf55d8-n4d68_6fc09a83-a593-4e4e-b4c0-4921265e7ac1/payload-pre-processing/0.log" Nov 02 10:44:02.618030 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:02.618001 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8hd89_565aea70-fb40-497b-9a8a-876abb6edb02/serve-healthcheck-canary/0.log" Nov 02 10:44:03.344368 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:03.344342 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cs7kk_0ff82d1d-e5cc-4703-847e-1c62ea27ef2e/kube-rbac-proxy/0.log" Nov 02 10:44:03.373287 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:03.373256 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cs7kk_0ff82d1d-e5cc-4703-847e-1c62ea27ef2e/exporter/0.log" Nov 02 10:44:03.403714 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:03.403684 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cs7kk_0ff82d1d-e5cc-4703-847e-1c62ea27ef2e/extractor/0.log" Nov 02 10:44:03.513554 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:03.513527 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-x8mlt_cb0c1710-51cc-419a-84d3-f9e57666226c/kube-auth-proxy/0.log" Nov 02 10:44:03.622816 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:03.622729 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-d7szj_f2174c08-7034-41d6-b383-20e50a1298ee/istio-proxy/0.log" Nov 02 10:44:03.639744 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:03.639717 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fgc89_cc4a7396-fc1a-40d9-9500-19ecb03d23b8/istio-proxy/0.log" Nov 02 10:44:03.657539 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:03.657495 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-l4czw_16563487-59d4-4d8a-ad4d-e4d2c9fa557c/istio-proxy/0.log" Nov 02 10:44:03.780265 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:03.780238 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dbbcf55d8-n4d68_6fc09a83-a593-4e4e-b4c0-4921265e7ac1/payload-pre-processing/0.log" Nov 02 10:44:04.054128 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:04.054057 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-597b95cb84-ffv4m_3c5b0649-6d82-4523-bf72-118b0529d31a/cert-manager-controller/0.log" Nov 02 10:44:04.071981 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:04.071955 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-457dp_072580f0-95f2-4082-b877-c0134f0e1a26/cert-manager-cainjector/0.log" Nov 02 10:44:04.086846 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:04.086820 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-ztmqw_ba2ac63a-0ce6-428c-be4e-2f16c0cbb437/cert-manager-webhook/0.log" Nov 02 10:44:04.573565 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:04.573538 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-48lmt_3427c5a6-d489-47b8-b398-4f18f09286e7/manager/0.log" Nov 02 10:44:04.590779 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:04.590756 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-48lmt_3427c5a6-d489-47b8-b398-4f18f09286e7/copy-manifests/0.log" Nov 02 10:44:05.181108 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:05.181080 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f755f6879-xh4h5_30722de2-3a23-49f9-9672-50446962e7e2/manager/0.log" Nov 02 10:44:05.727684 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:05.727645 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b555b5c85-sr2mh_c8787154-b48a-4aba-8c76-d1d9d0e434a2/manager/0.log" Nov 02 10:44:05.742575 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:05.742551 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2htfm_eadb4244-9782-409a-b4d2-e875d845c023/server/0.log" Nov 02 10:44:06.307253 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:06.307219 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-48lmt_3427c5a6-d489-47b8-b398-4f18f09286e7/manager/0.log" Nov 02 10:44:06.336659 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:06.336602 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-48lmt_3427c5a6-d489-47b8-b398-4f18f09286e7/copy-manifests/0.log" Nov 02 10:44:06.427248 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:06.427217 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d2kkp_77449e74-83c7-4592-bea4-48a7d20412d8/manager/0.log" Nov 02 10:44:06.479247 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:06.479222 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7shdp_925f1379-43e8-4594-ae67-ae2cf3bafab3/proxy/0.log" Nov 02 10:44:07.302991 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:07.302965 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-x8mlt_cb0c1710-51cc-419a-84d3-f9e57666226c/kube-auth-proxy/0.log" Nov 02 10:44:07.411720 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:07.411689 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-d7szj_f2174c08-7034-41d6-b383-20e50a1298ee/istio-proxy/0.log" Nov 02 10:44:07.431720 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:07.431674 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fgc89_cc4a7396-fc1a-40d9-9500-19ecb03d23b8/istio-proxy/0.log" Nov 02 10:44:07.450668 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:07.450643 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-l4czw_16563487-59d4-4d8a-ad4d-e4d2c9fa557c/istio-proxy/0.log" Nov 02 10:44:07.571119 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:07.571088 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dbbcf55d8-n4d68_6fc09a83-a593-4e4e-b4c0-4921265e7ac1/payload-pre-processing/0.log" Nov 02 10:44:08.007383 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:08.007315 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-df694_17892573-e467-4794-ae0a-937c6c472028/jobset-operator/0.log" Nov 02 10:44:08.045467 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:08.045444 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-x8mlt_cb0c1710-51cc-419a-84d3-f9e57666226c/kube-auth-proxy/0.log" Nov 02 10:44:08.150506 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:08.150474 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-d7szj_f2174c08-7034-41d6-b383-20e50a1298ee/istio-proxy/0.log" Nov 02 10:44:08.167569 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:08.167547 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-fgc89_cc4a7396-fc1a-40d9-9500-19ecb03d23b8/istio-proxy/0.log" Nov 02 10:44:08.186011 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:08.185969 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-l4czw_16563487-59d4-4d8a-ad4d-e4d2c9fa557c/istio-proxy/0.log" Nov 02 10:44:08.282469 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:08.282400 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dbbcf55d8-n4d68_6fc09a83-a593-4e4e-b4c0-4921265e7ac1/payload-pre-processing/0.log" Nov 02 10:44:08.734548 ip-10-0-138-234 kubenswrapper[4467]: I1102 10:44:08.734516 4467 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f755f6879-xh4h5_30722de2-3a23-49f9-9672-50446962e7e2/manager/0.log"